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ABSTRACT 


Human  performance  in  spatial  orientation  tasks  is  mainly  determined  by 
spatial  awareness  and  the  skills  to  transition  from  the  current  spatial  attitude  into 
the  desired  spatial  orientation  and  position.  Erroneous  spatial  awareness  may 
lead  to  degraded  task  performance,  to  the  loss  of  equipment,  to  serious  injuries, 
or  fatal  aviation  mishaps. 

The  use  of  UAVs  is  considered  beneficial  due  to  the  reduction  in  risk  to 
the  human  carrying  out  the  “mission”.  However,  the  remote  execution  of  such  a 
mission  is  extremely  demanding  for  the  operator.  If  extensive  use  of  UAVs  is  to 
become  routine,  a  number  of  concerns  that  may  influence  their  effective  use 
needs  to  be  addressed.  When  we  consider  the  human-in-the-loop  (HITL),  then 
vehicle  control  and  the  use  of  autonomy  are  important  issues  for  the  end  user. 

Therefore  this  thesis  will  investigate  the  use  of  a  virtual  avatar  in  the  flight 
simulator  software  (Weber  Box)  and  conduct  experimental  proof  of  concept 
(conduct  of  experiments  and  analysis,  evaluation  and  validation  of  the  data  of  the 
concept  using  actual  flight  simulation  software).  Results  of  a  study  (conducted  by 
Weber,  2006)  indicated  that  the  proposed  design  (Weber  Box)  seemed  to 
strongly  support  spatial  awareness  in  3D  orientation  tasks.  Time  to  assess  a 
spatial  situation  decreases  significantly,  whereas  accuracy  of  this  spatial 
judgment  at  least  maintains  its  level. 

This  study  investigated  human  orientation  performance  in  relation  to 
display  designs  that  support  mental  models  of  the  user’s  spatial  situation  under 
varying  workload  conditions.  The  main  goal  is  to  support  the  pilot/operator  with 
intuitive,  3D-based  information  which  improves  theirs  spatial  awareness  and 
supports  their  mental  model  of  spatial  position,  he/she  is  operating  under,  even 
with  varying  workload  conditions.  As  a  follow-up  study  has  to  be  identified, 
determining  whether  varying  workload  affects  performance  between  the  two 
display  designs,  and  if  there  is  a  significant  difference  to  a  set  of  properties  which 
are  essential  for  linking  virtual  avatars  and  spatial  awareness. 
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I.  INTRODUCTION 


A.  OVERVIEW 

Most  aircraft  disasters  are  the  result  of  inappropriate  action  or  actions 
carried  out  by  the  pilots  involved;  a  process  known  as  "pilot  error."  Aircraft 
disasters  are  usually  the  outcome  of  a  chain  of  events  where  many  of  the  factors 
involved  are  insignificant  in  their  own,  but  when  combined  with  each  other  or  with 
other  factors  they  can  result  in  a  situation  whereby  the  pilots  involved  become 
confused  as  to  what  is  happening  (a  loss  of  situational  awareness)  and  they 
begin  to  react  in  a  way  that  the  situations  end  up  with  catastrophic  results 
(Paterson,  2000). 

A  significant  causal  factor  as  to  why  pilots  become  confused  and  begin  to 
mishandle  the  situation  in  which  they  find  themselves,  is  among  many  others 
poor  cockpit  design.  Simple  visual  cues  increase  human  awareness  and 
perception  and  decrease  reaction  times.  Humans  are  visual  beings  requiring 
visual  cues  to  warn  them  of  impending  danger,  especially  in  combat  aviation.  The 
simplest  cues  are  those  that  allow  individuals  to  immerse  themselves  in  the 
situations  to  which  they  involve  and  must  respond.  Two-dimensional  (2-D) 
display  plan-views  of  aircraft  attitude  have  real  limits  on  what  types  of  information 
and  how  much  information  they  can  present  to  the  viewer  without  becoming 
disorienting  or  confusing.  Situational  Awareness  requires  a  transition  from  2-D  to 
3-D  display  perspectives. 

The  large  number  of  aircraft  accidents  due  to  spatial  disorientation  lead  to 
the  opinion  that  these  accidents  may  be  attributed  to  improper  instrument  use 
and  interpretation  (Clay,  1993).  The  need  to  fly  without  visual  references  was  an 
additional  incentive  to  improve  flight  instruments.  Technological  breakthroughs 
enabled  the  development  of  multifunctional  instruments  and  the  so  called  glass 
cockpit  (Mejdal  and  McCauley  and  Beringer,  2001). 
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The  basic  motivation  for  this  thesis  is  the  high  impact  of  spatial 
disorientation  (SD)  on  aviation  mishaps.  SD  has  become  a  major  issue  in 
modern  fixed  wing  and  rotor  wing  aviation  mishaps  prevention.  Mishaps  due  to 
spatial  disorientation  claim  nearly  three  times  more  lives  than  non-SD  mishaps 
(Matthews,  Previc,  and  Bunting,  2003). 

Studies  have  shown  that  maintaining  spatial  awareness  and  preventing 
spatial  disorientation  is  important  for  operating  Unmanned  Aerial  Vehicles 
(Matsangas,  2004).  Relationships  between  workload  and  operating  UAV 
performance  can  be  measured.  Results  of  a  study  conducted  by  Weber,  (2006) 
indicated  that  the  proposed  design  (Weber  Box)  seemed  to  sustain  remarkably 
well  the  spatial  awareness  in  3-D  orientation  tasks.  Time  to  assess  a  spatial 
situation  decreases  significantly,  with  no  loss  in  accuracy  of  this  spatial  judgment 
at  least  maintains  its  level.  Judgment  errors  were  minimized  and  the  extreme 
errors  were  almost  eliminated. 

The  purpose  of  the  current  study  was  to  assess  both  subjective  workload 
and  associated  operating  UAV  control  performance  under  varying  workload 
conditions  for  both  experienced  and  novice  operators/pilots.  Specifically,  the 
focus  was  to  assess  the  relationship  between  subjective  workload  and  various 
objective  measures  of  simulated  operation/performance  both  with  and  without  the 
introduction  of  a  secondary  task  involving  operating  more  than  one  UAV 
simultaneously.  Two  experiments  were  conducted  to  investigate  how  the 
proposed  design  affects  operators’  orientation  performance  and  to  test  the  new 
effectiveness  of  the  design  concept  under  varying  workload  conditions. 

B.  THESIS  ORGANIZATION 

Chapter  II  reviews  literature  covering  the  major  concepts,  issues  and 
systems  underlying  Situation  Awareness,  spatial  awareness  and  spatial 
disorientation.  The  methods  used  are  presented  in  Chapter  IV.  Chapter  V  covers 
the  analytical  strategy  and  presents  the  statistical  results.  Finally,  conclusions 
and  recommendations  for  future  research  are  offered  in  Chapter  VI  and  VII 
respectively. 
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II.  LITERATURE  REVIEW 


A.  SITUATION  AWARENESS 

Situation  Awareness  (SA)  is  a  term  used  originally  in  the  aircraft 
community  and  achieving  it  is  perhaps  the  one  of  the  most  difficult  aspects  of  an 
operator’s  work.  It  has  been  defined  by  Endsley  (1988)  as:  “...the  perception  of 
the  elements  within  a  volume  of  time  and  space,  the  comprehension  of  their 
meaning,  and  the  projection  of  their  status  in  the  near  future.”  It  is  therefore  clear 
that  SA  is  necessary  in  order  for  people  to  perform  tasks  effectively  (Endsley, 
2000).  Dominguez  (1994)  gives  a  similar  definition,  but  places  more  emphasis  on 
the  impact  of  awareness  on  cue  extraction  and  directed  perception  (i.e.  its 
contribution  to  attention).  For  Dominguez  (1994)  then,  SA  constitutes  a 
continuous  extraction  of  environmental  information,  integration  of  this  knowledge 
to  form  a  coherent  mental  picture,  and  the  use  of  that  mental  picture  in  directing 
further  perception  and  anticipating  future  events. 

Situation  Awareness  is  most  frequently  defined  in  operational  terms. 
Situation  Awareness  is  knowing  what  is  going  on  around  you.  We  have  been 
concerned  mainly  with  people  who  need  Situation  Awareness  for  specific 
reasons,  rather  than  individuals  who  has  been  largely  outside  the  scope  of 
human  factors  design  efforts.  For  a  given  operator,  therefore,  SA  is  defined  in 
terms  of  the  goals  and  decision  tasks  for  that  job.  The  pilot  does  not  need  to 
know  everything,  but  he/she  does  need  to  know  at  least  the  information  to 
achieve  a  safe  flight  whether  he/she  is  flying  an  aircraft  or  he/she  operates  a 
UAV  as  the  related  goal  respectively.  Although  the  "elements"  of  Situation 
Awareness  vary  widely  between  domains,  the  nature  of  SA  and  the  mechanisms 
used  for  achieving  SA  can  be  described  generically. 

Shown  in  Figure  1  this  definition  helps  to  establish  what  "knowing  what  is 
going  on"  means. 
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^ *  System  Capab  Irtv'^ 

•  Interface  Design 

•  Stress  &  Workload 

•  Complexity 

•  Autoncrton 


Task/System  Factors 


The  enhancement  of  operator  Situation  Awareness  has  become  a  major 
design  goal  for  those  developing  operator  interfaces,  automation  concepts  and 
training  programs  in  a  wide  variety  of  fields. 

Today's  systems  are  designed  and  therefore  capable  of  producing  a  huge 
amount  of  data,  both  on  the  status  of  their  own  components,  and  on  the  status  of 
the  external  environment.  Due  to  achievements  in  various  types  of  data  link  and 
communications,  systems  can  also  provide  data  on  almost  anything  anywhere  in 
the  world.  The  problem  with  today's  systems  is  not  a  lack  of  information,  but 
finding  what  is  needed  when  it  is  needed  fulfilling  the  principals  “least  to  know” 
and  “need  to  know”. 

Unfortunately,  there  is  a  gap  between  the  amount  of  data  being  produced 

and  disseminated  and  the  people's  ability  to  find  the  information  that  is  needed 
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and  process  it  with  the  rest  of  the  arriving  information  and  the  actual  information 
that  is  required  for  their  decisions.  This  information  must  be  integrated  and 
interpreted  correctly  as  well.  Issues  of  automation  and  "intelligent  systems"  have 
frequently  only  exacerbated  the  problem,  rather  than  aided  it  (Endsley  and  Kiris, 
1995;  Sarter  and  Woods,  1995).  In  addition  to  designing  systems  that  provide  the 
operator  with  the  needed  information  and  capabilities,  we  must  also  insure  that  it 
is  provided  in  a  way  that  is  useable  cognitively  as  well  as  physically.  We  want  to 
know  how  well  the  system  design  supports  the  operator's  ability  to  get  the 
needed  information  under  dynamic  operational  constraints  (i.e.  How  well  does  it 
bridge  the  information  gap?).  This  design  objective  and  measure  of  merit  has 
been  termed  “Situation  Awareness.”  As  an  expansion,  the  previous  general 
definition  of  SA  (Endsley,  1988)  has  been  found  to  be  applicable  across  a  wide 
variety  of  domains  as  discussed  below: 

1.  Level  1  SA  -  Perception 

In  this  first  degree  of  SA,  the  perception  of  the  elements  in  the 
environment,  humans  achieve  SA  by  perceiving  the  status,  characteristics,  and 
dynamics  of  all  relevant  elements  in  the  environment.  For  instance,  pilots  have  to 
perceive  important  elements  such  as  status  of  the  own  aircraft,  other  air  traffic, 
terrain,  flight  parameters  and  warnings. 

First,  perception  of  cues  (Level  1  SA)  is  fundamental.  Without  basic 
perception  of  important  information,  the  odds  of  forming  an  incorrect  picture  of 
the  situation  increase  dramatically.  Jones  and  Endsley  (1996)  found  that  76%  of 
SA  errors  in  pilots  could  be  traced  to  problems  in  perception  of  needed 
information  (due  to  either  failures  or  shortcomings  in  the  system  or  problems  with 
cognitive  processes).  In  military  aviation,  the  number  of  tasks  and  the  rate  of 
change  of  the  environment  is  very  high  because  of  the  dynamic  flight  maneuvers, 
ground-  and  airborne-enemy  activities,  sophisticated  weapon  systems  etc. 

2.  Level  2  SA  -  Comprehension 

The  second  level  of  SA  is  comprehension  of  the  current  situation  and  is 
based  on  synthesis  Level  1  elements.  The  key  for  Level  2  SA  is  that  the  human 


5 


must  integrate  Level  1  information  to  form  a  complete  mental  representation  of 
the  environment.  Situation  Awareness  as  a  construct  goes  beyond  mere 
perception  however.  It  also  encompasses  how  people  combine,  interpret,  store, 
and  retain  information.  Thus,  it  includes  more  than  perceiving  or  attending  to 
information,  but  also  the  integration  of  multiple  pieces  of  information  and  a 
determination  of  their  relevance  to  the  person's  goals  (Level  2  SA). 

This  is  analogous  to  having  a  high  level  of  reading  comprehension  as 
compared  to  just  reading  words.  Twenty  percent  of  SA  errors  were  found  to 
involve  problems  with  Level  2  SA  (Jones  and  Endsley,  1996).  Flach  (1995)  points 
out  that  "the  construct  of  Situation  Awareness  demands  that  the  problem  of 
meaning  be  tackled  head-on.  Meaning  must  be  considered  both  in  the  sense  of 
subjective  interpretation  (awareness)  and  in  the  sense  of  objective  significance  or 
importance  (situation)."  A  person  with  Level  2  SA  has  been  able  to  derive 
operationally  relevant  meaning  and  significance  from  the  Level  1  data  perceived. 

3.  Level  3  SA  -  Projection 

The  third  level  of  SA,  the  projection  of  future  status,  is  the  highest  level  of 
SA.  It  enables  a  person  to  predict,  at  least  in  the  very  near  future,  the  evolving 
situation.  This  is  achieved  through  excellent  knowledge  of  the  functioning  and 
dynamics  of  the  system  and  a  comprehension  of  the  situation  (both  Level  1  and 
Level  2  SA).  Pilots  have  to  constantly  project  their  current  situation  into  the  future 
and  will  base  their  decisions  on  this  prediction. 

At  the  highest  level  of  SA,  the  ability  to  forecast  future  situation  events  and 
dynamics  (Level  3  SA)  marks  operators  who  have  the  highest  level  of 
understanding  of  the  situation.  This  ability  to  project  from  current  events  and 
dynamics  to  anticipate  future  events  (and  their  implications)  allows  for  timely 
decision  making. 

4.  SA  as  a  Multidimensional  Construct 

The  fact  that  SA  is  not  a  single  ability  creates  serious  problems  for  human 
factors  researchers  because  it  means  they  have  no  single  measure  of  the 
phenomenon  in  which  they  are  most  interested.  Indeed  when  Dennehy  and 
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Deighton  (1997)  reviewed  the  literature  from  the  period  of  1979  to  1992,  they 
identified  no  less  than  28  different  variables  which  had  been  used  as  measures 
of  SA.  Considering  that  these  variables,  Dennehy  and  Deighton  set  out  to  find 
the  main  SA  subscales  or  "meta-categories,"  as  follows: 

•  SA  Meta-Category  1  -  Pilot  Knowledge:  The  first  principal 
component  was  a  composite  measure  of  knowledge,  per  se. 

•  SA  Meta-Category  2  -  Anticipation  and  Understanding  of  Future 
Events:  The  second  principal  component  was  a  composite  measure 
of  an  ability  cluster  for  anticipating  and  understanding  future  events. 

•  SA  Meta-Category  3  -  Capacity  to  Manage  Stress,  Effort,  and 
Commitment:  The  third  principal  component  was  a  composite 
measure  of  an  ability  cluster  for  managing  stress  and  maintaining 
effort  and  commitment. 

•  SA  Meta-Category  4  -  Capacity  to  Perceive,  Attend,  Assess,  and 
Assimilate  Information:  The  fourth  principal  component  was  a 
composite  measure  of  an  ability  cluster  for  obtaining  and 
processing  information  from  the  world  itself,  both  directly  and  from 
the  cockpit  dials  and  gauges  available. 

•  SA  Meta-Category  5  -  Overall  Awareness:  The  fifth  principal 
component  was  a  composite  measure  of  general  awareness, 
similar  to  the  "g-factor"  popular  among  some  intelligence  theorists. 

In  a  further  attempt  to  understand  this  clustering,  Dennehy  and  Deighton 
adopted  the  Endler-Mischel  "Interactionist  Theory"  (Endler,  1973;  Mischel,  1973). 
This  theory  holds  that  it  is  impossible  "to  understand  the  role  of  an  operator  in 
isolation  from  the  context  within  which  she/he  operates"  (Dennehy  and 
Deighton).  The  result  is  a  "P-E  Fit"  model,  one  which  tries  to  "fit"  the  person  into 
his/her  environment.  Dennehy  and  Deighton  characterized  SA  as  the 
"operational  space"  provided  by  the  two  interacting  domains  in  Figure  2. 
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Figure  2.  The  Interactionist  ("P-E  Fit")  Model  of  Situational  Awareness  (From 

Dennehy  and  Deighton  1997) 

However,  Endsley  (1995,  1999)  points  out  that  it  is  simultaneously 
possible  to  divide  up  a  pilot's  declarative  knowledge  base  (Dennehy  and 
Deighton's  first  meta-category)  according  to  the  type  of  knowledge  involved,  as 
follows: 

•  Geographical  SA:  This  knowledge  domain  stores  facts  about  the 
physical  world,  for  example,  the  heights  of  mountains,  the  nature  of 
the  terrain,  the  locations  of  alternative  airfields,  the  direction  of  the 
prevailing  wind,  etc. 

•  Spatial/Temporal  SA:  This  knowledge  domain  stores  facts  about 
the  four-dimensional  physical  world  (the  three  spatial  dimensions, 
plus  time),  for  example,  aircraft  position  and  speed,  time  into 
mission,  etc.  This  is  the  most  important  for  the  current  study. 

•  System  SA:  This  knowledge  domain  stores  facts  about  system 
states,  for  example,  engine,  control,  or  instrument  malfunctions, 
fuel  status,  etc. 

•  Environmental  SA:  This  knowledge  domain  stores  facts  about 
weather  and  visibility,  together  with  such  things  as  restricted  (keep 
out)  or  prescribed  (keep  in)  airspace. 
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•  Tactical  SA;  This  knowledge  domain  stores  facts  about  the  identity 
and  capabilities  of  all  other  units  in  the  vicinity  (and,  if  military,  that 
will  include  their  combat  intention  as  well). 

Endsley  brought  all  these  concepts  together  into  what  he  presents  as  an 

all-embracing  theoretical  model  of  SA  (Figure  3). 
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Figure  3.  Model  of  Situational  Awareness  (From  Endsley  1995,  1999) 

Another  approach  is  the  so-called  situated  cognition  model  by  Miller  and 
Shattuck  (2004),  which  describes  SA  model  as  a  process,  rather  than  certain 
fixed  states  as  in  Ensley’s  model.  It  is  based  on  the  one  hand,  on  Pew’s  model  of 
ideal  SA,  achievable  SA  and  actual  SA  (Pew,  2000),  which  describes  the 
problem  of  measuring  SA.  On  the  other  hand,  it  includes  the  lens-model  of 
Tucker  and  Flammond,  of  information  selection,  and  filtering  (Tucker,  1964). 
Miller  and  Shattuck’s  model  focuses  on  processes  rather  than  states  and 
includes  both  human  and  machine  elements  of  a  system.  It  is  oriented  toward 
assessing  human-system  performance,  tracks  the  evolution  of  activities  and 
cognition,  and  links  it  to  cognitive  decision  processes.  From  this  perspective  it 

describes  the  process  of  building  SA  starting  at  the  ground  truth,  which  is 
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selectively  perceived  by  different  layers  of  sensors  (lenses).  Every  lens  filters  and 
transforms  the  received  information.  Then,  the  filtered  information  is  processed 
by  methods  similar  to  Endsley’s  3-Levels-of-SA  model.  In  Miller  and  Shattuck’s 
model,  these  levels  are  not  independent  levels  of  SA  (Figure  4).  They  are,  rather, 
specific  cognitive  processes  which  map  additional  aspects  of  the  task  and  the 
environment  like  goals,  expectations,  personality  etc.  to  the  originally  perceived 
information  (Miller  and  Shattuck,  2004). 


Dynamic  Mode)  of  Situated  Cognition 


2004) 


SA  requires  an  operator  to  “quickly  detect,  integrate  and  interpret  data 
gathered  from  the  environment.”  In  real-world  conditions.  Situational  Awareness 
is  hampered  by  two  factors:  data  spread  throughout  the  visual  field  and  data  that 
is  noisy  (Green,  Odom  and  Yates,  1995).  Understanding  how  data  flow  through 
the  system,  where  the  data  may  have  been  blocked,  and  how  lenses  may  be 
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skewed  is  vital  to  knowing  how  to  redesign  systems  and  develop  appropriate 
training  (Miller  and  Shattuck,  2004).  One  of  the  greatest  challenges  facing  the 
operators  and  technology  providers  is  to  match  our  increasing  capability  to 
gather,  process,  and  display  Situational  Awareness  information  with  a 
corresponding  increase  in  our  ability  to  use  that  information  (Marsh,  2000). 
Increased  availability  of  information  about  the  situation  may  also  be 
counterproductive  with  respect  to  time  critical  decision  making.  When  decision 
makers  expect  information  to  be  sparse,  they  tend  to  rely  on  judgment  and 
experience  to  fill  the  voids.  When  information  is  massive  and  confusing,  decision 
makers  may  often  delay  taking  action  until  additional  information  can  be  gathered 
to  fill  voids  or  resolve  ambiguities.  They  may  reorient  their  decision  making 
process  from  judgment  based  on  experience  to  reliance  on  detailed  analysis 
based  on  hard  data.  The  result  can  be  “analysis  paralysis,”  leading  to  a  delay  of 
the  decision  until  it  is  too  late  (Marsh,  2000).  Designers  can  help  avoid  these 
pitfalls  by  using  properly  designed  technology  coupled  with  well-trained 
operators,  which  will  result  in  optimal  human-system  performance  (Miller  and 
Shattuck,  2004). 

5.  How  Do  We  Get  SA? 

SA  is  the  product  of  all  various  sources  of  information,  as  shown  in  Figure 
5.  Cues  may  be  received  through  visual,  aural,  tactile,  and  olfactory  or  taste 
receptors.  As  we  move  towards  the  instantiation  of  remote  operators  in  many 
domains  (e.g.,  unmanned  air  vehicles,  remote  maintenance,  etc.),  a  major 
challenge  will  be  providing  sufficient  information  through  a  remote  interface  to 
compensate  for  the  cues  once  perceived  directly  (Endsley,  1995). 
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Figure  5.  Sources  of  SA  Information  (From  Endsley  1995c,  1997) 

It  is  important  to  note  that  there  is  a  tendency  to  focus  on  the  information 
provided  through  the  system  and  its  operator  interface  which  is  not  the  only 
source  of  SA.  In  many  domains,  operators  may  be  able  to  directly  view  and  hear 
information  from  the  environment  itself,  although  in  some  cases  they  may  not.  It 
is  important  therefore  that  analyses  of  the  SA  provided  by  system  designs  also 
take  into  account  that  information  that  operators  derive  from  other  means.  That 
is,  what  is  the  value  added  (or  subtracted)  by  a  given  system  taking  into  account 
what  information  one  already  gets  via  other  means  and  what  it  may  cost  in  terms 
of  interference  with  that  information  (Endsley  2000).  Of  all  the  data  the  system 
possesses,  some  portion  is  displayed  to  the  operator  via  its  user  interface.  Of  this 
information,  the  operator  perceives  and  interprets  some  portion,  resulting  in  SA. 
The  role  of  others  in  the  process  of  developing  SA  has  also  received  attention. 
Verbal  and  non-verbal  communication  with  others  (including  radio 
communication,  hand  signals  and  "wing-tipping"  by  other  pilots)  has  historically 
been  found  to  be  an  important  source  of  SA  information  (Endsley  2000). 
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B.  SPATIAL  AWARENESS 

1.  Spatial  Awareness 

To  gain  a  better  understanding  of  the  term  spatial  awareness,  we  define 
its  meaning  and  other  terms  used  in  a  similar  context.  Spatial  Awareness  is  best 
understood  as  an  important  component  of  Situational  Awareness.  In  a  flight 
environment,  developing  Situation  Awareness  involves  an  assessment  of 
numerous  factors  both  internal  and  external.  In  order  to  identify  the  presence, 
magnitude,  and  possible  intentions  of  a  perceived  threat,  the  operator  must 
assess  various  aspects  of  aircraft  behavior  such  as  range,  heading,  altitude, 
speed,  attitude  (climbing  or  descending)  and  location  on  earth.  In  order  to 
evaluate  alternative  responses  to  the  potential  threat  the  operator  must  take  into 
account  the  status  of  defensive  assets,  considering  factors  that  may  limit 
available  options,  such  as  equipment  malfunction  or  damage,  fuel  availability, 
atmospheric  conditions,  and  terrain.  All  of  these  evaluations  are  made  within  a 
constantly  changing  environment  and  under  highly  stressful  situations  in  which 
timing  is  critical  and  errors  may  be  catastrophic. 

Spatial  awareness  refers  to  an  operator’s  comprehension  of  the  3D 
geometry  of  the  environment  in  which  he/she  is  operating.  Three-dimensional 
information  contained  in  the  environment  includes  the  absolute  distance  of 
objects  (distance  from  an  observer  to  an  object),  the  relative  distance  of  objects 
(distance  between  one  object  and  another  object  or  the  distance  between 
different  parts  of  a  single  object),  and  the  true  3D  shape  of  objects  (Wickens, 
Todd  and  Seidler,  1989b).  In  a  flight  environment,  this  information  is  available  to 
an  operator  directly  from  his  forward  field  of  view  and  other  senses  and  from 
visual  displays  (Endsley,  1988).  In  some  instances,  for  example  at  night  or  when 
objects  are  out  of  viewing  range,  the  operator  may  have  to  rely  solely  on  visual 
displays  for  a  spatial  representation  of  the  environment  (Andre,  Wickens, 
Moorman  and  Boschelli,  1991).  Visual  displays  are  therefore  critical  to  operators’ 
spatial  awareness. 
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Currently,  most  operators  rely  on  plan-view  displays  to  develop  mental 
models  of  the  space  in  which  they  are  operating.  A  major  limitation  of  this  type  of 
display  is  that  it  can  only  represent  information  from  two  dimensions  of  space; 
the  vertical  dimension  is  typically  encoded  in  a  textual  format.  To  obtain 
information  about  the  vertical  dimension  of  a  track  in  the  environment,  operators 
are  required  to  “hook”  the  track  and  press  a  button  to  obtain  textual  readouts  of 
altitude.  To  determine  aircraft  attitude,  operators  must  monitor  altitude  readouts 
over  time  and  observe  changes.  Operators  are  therefore  forced  to  integrate 
textual  with  spatial  information  and  mentally  reconstruct  the  3D  nature  of  the 
visual  scene.  This  process  requires  valuable  cognitive  resources  and  decision¬ 
making  time  (Haskell  and  Wickens,  1993).  These  limitations  of  plan-view  displays 
may  be  overcome  by  3D  display  technology.  Three-dimensional  displays  can 
depict  all  three  dimensions  of  space  in  a  completely  spatial  format  thereby 
eliminating  the  requirement  to  integrate  textual  with  spatial  information.  All  of  this 
information  is  contained  within  a  single  display  which  reduces  the  need  for 
mental  integration  of  information  from  multiple  sources  (Woods,  1984).  Three- 
dimensional  displays  also  provide  a  more  natural  or  ecological  representation  of 
the  “real  world”  (Wickens  et  al.,  1989b).  Three-dimensional  computer  graphics 
systems  include  perspective  displays,  stereoscopic  displays,  rotating  displays, 
head-motion  tracking  displays,  holographic  displays,  and  multi-plane  displays. 

Ellis,  McGreevy  and  Hitchcock  (1984,  1987)  examined  how  pilots’ 
avoidance  maneuvers  varied  as  a  function  of  display  type:  plan-view  versus 
perspective.  Results  from  10  airline  pilots  showed  that  the  perspective  display 
produced  improved  avoidance  maneuvering;  pilots  took  less  time  to  identify 
collision  hazards  and  recommend  a  maneuver,  fewer  errors  were  made  in 
selecting  a  maneuver.  Pilots  were  twice  as  more  likely  to  select  a  vertical 
maneuver  with  the  perspective  display  probably  due  to  the  more  natural 
presentation  of  vertical  separation.  Bemis,  Leeds  and  Winer  (1988)  compared  a 
plan-view  with  a  perspective  display  for  the  task  of  detecting  an  airborne  threat 
and  selecting  the  closest  friendly  aircraft  to  intercept  the  threat.  The  results  from 
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21  naval  operational  personnel  showed  that  fewer  errors  in  detecting  threats 
were  made  in  the  perspective  display  condition.  The  subjects  were  also  more 
accurate  and  quicker  at  intercepting  aircraft  using  the  perspective  display.  Survey 
results  showed  that  19  of  the  21  subjects  preferred  the  perspective  display. 

2.  Models  of  Spatial  Awareness 

According  to  the  common  definition  of  SA,  in  a  three-dimensional  (3D) 
orientation  task,  a  person  who  has  good  SA  has  fast  access  to  an  accurate 
mental  representation  of  the  altering  environment  and  may  be  able  to  predict  his 
spatial  situation  in  the  near  future.  For  example,  an  aviator  with  excellent  SA  may 
not  be  consciously  thinking  about  the  fact  that  there  is  an  aircraft  nearby.  He  is 
able  to  operate  the  aircraft  proactively  and  correctly  according  to  this  situation. 
The  pilot  will  do  so  quickly  and  precisely  based  on  the  ability  to  rapidly  access 
the  information  from  memory  (Wickens,  2002).  Thus,  instrument  design  that 
supports  SA  will  facilitate  the  appropriate  reaction  in  uncertain  situations. 

Spatial  awareness  can  be  seen  as  a  subset  of  Situation  Awareness  or 
rather  a  specific  application  of  it  and  it  has  many,  often  vague  definitions.  For  this 
study,  we  consider  spatial  awareness  as  an  extension  of  spatial  orientation, 
where  spatial  orientation  is  understood  as  the  ability  to  recognize  an  object’s 
orientation  or  position  in  three-dimensional  space.  It  describes  spatial  awareness 
as  the  knowledge  of  their  position  relative  to  the  desired  flight  route  and  the 
ground  and  its  spatial  attitude.  In  addition  it  includes  the  knowledge  about  the 
current  flight  maneuver;  its  own  spatial  behavior. 

Wickens  (2002)  pointed  out  that  a  pilot’s  spatial  awareness  is  determined 
by  six  crucial  variables:  pitch,  roll  and  yaw  (slip)  of  the  aircraft,  altitude,  deviation 
from  a  flight  path,  and  position  along  a  flight  path.  More  factors  directly  and 
indirectly,  need  to  be  continuously  monitored  to  build  a  complete  spatial  model. 
Two  of  the  most  obvious  are  airspeed  and  stall-indicator.  In  addition,  knowledge 
is  needed  about  other  air  traffic,  ground  and  weather.  All  variables  are  cross- 
linked,  which  means  that  one  or  more  variables  influence  other  variables  in  the 
future.  For  instance  the  combination  pitch  and  airspeed  may  lead  to  later  stalling 
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of  the  aircraft.  Skilled  pilots  represent  this  linkage  in  a  mental  model  and 
implement  time  constrains,  latencies,  and  additional  information  within  this 
model. 

3.  Measuring  Situational  and  Spatial  Awareness 

An  individual’s  SA  is  a  relatively  difficult  construct  to  measure  (Hancock 
and  Desmond,  2001).  Another  issue  is  to  determine  how  much  SA  is  sufficient  for 
the  task.  Endsley  states  that  SA  and  performance  measures  are  only  linked 
probabilistically,  and  that  there  are  no  set  thresholds  of  SA  to  guarantee  a  given 
level  of  performance  (Endsley,  Sollenberger,  and  Stein,  2000).  Endsley  et  al. 
therefore  propose  relative  levels  of  SA.  In  terms  of  the  design  issue,  this  would 
mean  an  increase  in  SA  (after  the  addition  of  new  layout)  relative  to  the  level 
measured  prior  to  this  introduction.  Measurement  should  also  enable  comparison 
to  the  “ideal”  SA:  that  is,  perfect  knowledge  on  all  relevant  aspects  of  a  situation, 
with  no  gaps  or  holes  in  it.  As  we  discussed  earlier.  Pew  states  that  actual  SA 
tends  to  be  a  subset  of  the  achievable  SA,  which  is  again  a  subset  of  the  ideal 
SA  of  any  situation  (Pew,  2000). 

An  individual’s  level  of  SA  will  be  partially  determined  by  the  quantity  and 
quality  of  the  information  available  and  by  the  individual’s  ability  to  utilize  the 
important  information  sources  and  fill  in  for  missing,  imprecise,  or  incomplete 
information.  Possible  reasons  for  lowered  SA  are  environmental  factors  such  as 
lack  of  anticipation  of  vital  factors  (due  to  lack  of  awareness  of  the  importance  of 
these  factors,  or  attention  being  focused  on  other  things)  (Entin,  1998).  Entin 
refers  to  two  types  of  SA  measures;  high-level  and  detailed  SA.  High-level  SA 
measures  facts  according  to  subjects’  responses  to  general  questions  about  the 
current  situation.  The  detailed  SA  is  measured  with  questions  about  elements  of 
the  situation  based  on  a  broad-ranging  assessment  of  operator  SA  requirements) 
(Gawron,  2000).  Detailed  SA  reduces  when  information  changes  rapidly.  In 
aviation  tasks,  operators  may  not  have  time  to  integrate  all  the  incoming 
information  into  a  coherent  picture.  Basic  types  of  SA  measures  derive  from 
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Entin’s  study  of  detailed  and  high  level  SA.  One  is  based  on  the  responses  of 
subjects’  to  questionnaires  that  included  questions  designed  to  capture  specific 
elements  of  the  situation. 

However,  there  are  several  other  means  by  which  SA  can  be  measured, 
according  to  Pew  (2000),  which  fall  into  the  following  categories.  Direct  systems 
performance  measures  are  only  applicable  in  very  limited  situations.  Disruptions 
in  order  to  disorient  the  operator  can  be  introduced  (Pew  and  Mavor,  1998).  The 
operator’s  recovery  time  and  the  success  of  recovery  tactics  taken  can  be 
measured.  Performance  measures  are  often  not  sufficient  to  diagnose  human- 
system  relationships  in  detail.  In  many  cases  we  have  to  measure  subjective 
facts  as  mental  models  or  mental  workload  as  well  (Endsley  et  al.,  2003). 

Subjective  measures  include  self-assessments,  expert  judgments,  peer 
ratings,  and  supervisor  or  instructor  ratings.  A  different  way  to  measure  the 
gradient  of  change  in  SA  is  to  infer  it  from  objective  performance  measurements. 
In  our  opinion  objective  performance  measurement  should  be  conducted 
wherever  possible.  Subjective  methods  should  be  conducted  to  explain  data  of 
objective  measurements. 

Process  measures  are  considered  as  very  effective  in  measuring  SA. 
These  may  include  eye-tracing,  head  tracking,  information  acquisition,  flight 
parameter  analysis  and  so  on  (Endsley  et  al.,  2003). 

C.  SPATIAL  ORIENTATION  -  DISORIENTATION 

Spatial  disorientation  (SD)  is  the  most  common  cause  of  human-related 
aircraft  accidents.  SD  requires  the  knowledge  of  both  the  physiology  and 
psychology  of  the  human  in  flight  and,  to  a  lesser  extent  but  still  important,  an 
understanding  of  the  physics  of  an  aircraft  in  motion.  When  reading  about  an 
accident  involving  SD,  terms  like  visual  illusions,  vestibular  misperceptions,  task 
saturation,  weather,  motion,  and  aircraft  experience  are  commonly  found.  In  the 
past,  researchers  aimed  much  of  the  countermeasures  research  at  improving 
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their  understanding  of  these  misperceptions.  However,  in  order  to  better 
understand  the  overall  situation,  researchers  must  start  with  the  SD  definition 
and  arrange  each  condition  into  distinct  categories  (Figure  6). 

The  most  widely  used  and  accepted  general  definition  is  "A  state 
characterized  by  an  erroneous  sense  of  one's  position  and  motion  relative  to  the 
plane  of  the  earth's  surface"  (Freeman,  et  al,  1989-91).  Prior  to  this  accepted 
definition,  researchers  could  not  be  certain  that  a  particular  incident  qualified  as 
SD  or  as  another  phenomenon  altogether — often  masking  the  real  magnitude  of 
the  issue.  To  better  support  the  use  of  this  definition  by  researchers,  pilots, 
physicians,  and  physiologists,  an  operational  definition  recently  emerged — "An 
erroneous  sense  of  the  magnitude  or  direction  of  any  of  the  aircraft  control  and 
performance  flight  parameters."  (Gillingham,  1992). 


Figure  6.  Inner  ear  with  semicircular  canals  shown  likening  them  to  the  roll, 
pitch  and  yaw  axis  of  an  aircraft  (From  www.atlasaviation.com) 

Because  of  the  different  ways  SD  can  occur,  it  is  easier  to  study  SD  by 
separating  it  into  three  distinct  categories:  Type  I — unrecognized.  Type  II — 
recognized,  and  Type  III — incapacitating,  where  each  type  impacts  the  pilot  in  a 
different  way  (Heinie  and  Ercoline). 

The  first  group.  Type  I — unrecognized  SD,  explains  the  phenomenon  as  a 
state  where  the  pilot  is  unaware  of  the  flight  parameters  described  in  the 
operational  definition.  This  is  the  most  common  type  of  SD  due  to  many  psycho- 
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physiological  variables  (e.g.,  task  saturation,  canalized  attention,  fatigue,  etc.). 
Many  operators  state  that  this  as  a  simple  failure  of  the  pilot  to  maintain  an 
appropriate  instrument  crosscheck.  An  example  of  Type  I  SD  is  the  post-roll  or 
Gillingham  Illusion. 

The  second  group  of  SD  sensations  encompasses  those  incidents  that 
produce  a  recognized  phenomenon  known  as  a  sensory  mismatch  or  at  least  the 
awareness  that  something  has  gone  wrong.  This  is  labeled  as  Type  II — 
recognized  SD.  An  explanation  of  the  classic  Graveyard  Spin  Illusion 
demonstrates  Type  II  SD  (see  Figure  7). 


Figure  7.  Graveyard  Spin  Illusion  (From  www.atlasaviation.com) 

In  this  example,  the  pilot  enters  a  spin,  becomes  stabilized  in  yaw,  and 
realizes  a  need  to  move  the  controls  opposite  to  the  direction  of  rotation.  Once 
the  pilot  applies  the  opposite  controls,  the  aerodynamic  result  is  a  decrease  in 
the  aircraft's  angular  rotational  yaw  followed  by  a  false  sensation  of  the  aircraft 
beginning  to  spin  in  the  opposite  direction.  When  this  occurs  and  if  the  pilot  looks 
at  the  aircraft  turn  needle  or  compass  card,  the  pilot  experiences  a  sensory 
conflict.  The  turn  needle  will  indicate  a  turn  in  one  direction,  while  the  inner  ear 
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sensation  will  generate  a  feeling  that  the  aircraft  is  turning  in  the  opposite 
direction.  When  this  occurs,  the  pilot  often  suspects  an  instrument  malfunction 
and  does  not  recognize  the  situation  as  SD. 

The  third  and  last  type  of  SD  is  the  least  common.  Researchers  call  it 
Type  III — incapacitating  SD.  Few  studies  of  this  type  of  SD  exist,  but  researchers 
know  that  it  does  occur,  through  experience  and  pilot  reports.  An  example  of 
Type  III  SD  is  called  the  Giant  Hand  Illusion.  The  SD  phenomenon  has  been 
intertwined  with  aviation  since  the  beginning  of  manned  flight,  and  only  a 
concerted  and  coordinated  research  effort  will  make  a  difference  in  reducing  SD 
mishaps.  This  effort  begins  with  an  understanding  of  the  definition  of  SD,  along 
with  its  three  distinct  types  (Heinie  and  Ercoline). 

D.  HUMAN  PERFORMANCE  AND  COGNITION 

1.  Human  Perception  and  Cognition 

The  term  cognition  is  used  in  several  different  loosely  related  ways.  In 
psychology  it  is  used  to  refer  to  the  mental  processes  of  an  individual,  with 
particular  relation  to  a  view  that  argues  that  the  mind  has  internal  mental  states 
(such  as  beliefs,  desires  and  intentions)  and  can  be  understood  in  terms  of 
information  processing,  especially  when  a  lot  of  abstraction  is  involved,  or 
processes  such  as  knowledge,  expertise  or  learning,  for  example,  are  at  work. 
The  term  cognition  also  is  used  in  a  wider  sense  to  mean  the  act  of  knowing  or 
knowledge.  It  was  derived  from  psychological  science  which  attempted  to  explain 
human  behavior  and  reasoning  (Matthews,  Davies,  Westerman,  and  Stammers, 
2000).  Unlike  behaviorism,  cognitive  theory  focuses  on  what  is  going  on  inside 
the  person’s  mind.  Cognitive  learning  is  not  just  a  change  in  behavior;  it  is  a 
change  in  the  way  a  person  thinks,  understands,  or  feels. 

Perception  is  defined  as  a  direct  consequence  of  selective  attention,  which 
involves  the  extraction  of  meaning  from  an  information  set  processed  by  the 
human  senses  (Wickens  et  al.,  2004).  In  this  study  we  focus  on  higher  level 
perceptions  which  answer  questions  of  “What  am  I  in?”,  “Where  I  am?”,  “What  is 
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my  attitude?”,  “Where  am  I  going?”  and  so  on  (Warren  and  Wertheim,  1990). 
Hence,  we  start  with  thinking  at  a  level  above  human  sensory  systems  and 
physical  stimulus. 

Endsley  (1988;  1990;  1995)  proposed  a  framework  model  based  on 
information  processing  theory  (Wickens,  1992).  The  cognitive  mechanisms  that 
are  important  for  the  development  of  SA  are  shown  in  Figure  8. 
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Figure  8.  Mechanisms  and  Processes  involved  in  SA  (From  Endsley  1988; 

1990;  1995) 

Wickens  (1984)  hypothesized  that  human  attention  capacity  should  be 
conceived  as  multiple  resource  pools,  with  dual-task  interference  being  greatest 
when  tasks  compete  for  similar  processing  resources  and  least  (or  non-existent) 
when  tasks  draw  from  different  resource  pools. 

2.  Human  Performance  Models 

Human  performance  models  can  be  seen  in  three  major  tracks:  Manual 

Control  Models  of  human  control  in  closed-loop  systems.  Task  Network  Models 

are  concerned  with  fundamentally  predicting  the  probability  of  success  and 

performance  time,  and  Cognitive  Architectures  Models  are  typically  based  on 
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theories  of  human  performance  capacities  and  limitations  (Pew,  2000).  The  first 
two  types  of  models  arose  from  an  engineering  point  of  view  and  are  based  on 
control  theory.  Control  theory  describes  any  kind  of  system  that  has  a  feedback- 
loop. 

Cognition  and  human  performance  are  linked  in  cognitive  architecture 
models.  In  order  to  accomplish  this  we  have  to  address  more  factors  and 
connections.  Matthews  et  al.  (2000)  call  these  major  factors  the  “energetics  of 
cognition”  and  associated  with  it  capacity,  resources,  and  attention.  Capacity  is 
understood  as  the  physiological  and  physical  limits  of  human  performance. 
Resources  refer  to  a  limitation  of  attention.  We  might  not  have  enough  attention 
to  follow  many  parallel  things  that  are  happening  simultaneously.  Resource 
theory  is  widely  as  a  base  theory  to  study  attention.  Attention  has  selective  and 
intensive  characteristics.  The  selective  aspect  refers  to  a  choice  of  reaction  to  a 
certain  stimuli  rather  than  to  others  (Matthews  et  al.,  2000).  Divided  attention  is 
the  ability  to  react  to  two  things  or  more  at  the  same  time  or  to  accomplish  two  or 
more  tasks  or  mental  activities  (Wickens  et  al.,  2004). 

We  know  that  humans  are  not  able  to  focus  on  a  single  object  for  more 
than  some  seconds.  People  have  difficulty  maintaining  attention  over  a  longer 
period.  This  is  supported  by  various  studies  of  the  1940’s  and  50’s  which  did 
research  on  performance  of  radar  and  sonar  operators  and  their  detection  time 
and  error  rate  (Matthews  et  al.,  2000).  To  explain  sustained  attention,  four  major 
models  were  developed  over  time:  the  filter  theory,  the  expectancy  theory,  the 
arousal  theory  and  the  resource  theory  as  described  by  Matthews  et  al.  (2000). 

The  filter  theory  predicts  that  vigilance  tasks  in  which  signals  are  present 
only  for  a  short  period  will  contribute  to  a  faster  decrease  of  attention  than  tasks 
with  signals  that  are  present  throughout  a  longer  period. 

According  to  the  expectancy  theory  observers  keep  track  of  information  to 
extrapolate  the  occurrence  of  future  information.  Over  time,  the  observer’s 
internal  estimate  of  the  next  information  probability  changes. 
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The  basis  of  the  arousal  theory  is  that  sustained  attention  leads  to  a  lower 
level  of  arousal  of  the  central  nervous  system  (Matthews  et  al.,  2000).  It  was 
shown  by  some  experiments  that  cortical  arousal  level  decays  almost 
independently  from  the  character  of  the  task. 

Matthews  et  al.  (2000)  state  that  resource  theory  describes  experiments 
that  show  that  task  workload  plays  a  major  role  in  vigilance  decline.  This  theory 
suggests  that  performance  in  demanding  tasks  is  more  sensitive  to  resource 
limitations  by  fatigue  than  simpler  tasks.  This  position  is  supported  by  a  number 
of  single  and  dual-task  experiments  (Wickens  and  Gosney,  2003). 

3.  Working  Memory  and  Attention 

Several  factors  influence  the  accuracy  and  completeness  of  Situation 
Awareness  that  individual  operators  derive  from  their  environment.  First,  humans 
are  limited  by  working  memory  and  attention.  The  way  in  which  attention  is 
employed  in  a  complex  environment  with  multiple  competing  cues  is  essential  in 
determining  which  aspects  of  the  situation  will  be  processed  to  form  Situation 
Awareness.  Information  must  be  integrated  with  other  information,  compared  to 
goal  states  and  projected  into  the  future  -  all  heavily  demanding  on  working 
memory  (Endsley  2000).  Several  recent  studies  have  confirmed  the  role  of 
attention  in  Situation  Awareness.  Endsley  and  Smith  (1996)  found  that  fighter 
pilots'  attention  to  targets  on  a  tactical  situation  display  was  directly  related  to  the 
importance  of  those  targets  in  their  tactical  tasks. 

Attention  to  information  is  prioritized  based  on  how  important  that 
information  is  perceived  to  be  (Endsley  2000).  Even  experienced  operators  can 
make  errors  in  this  process,  neglecting  to  attend  to  certain  information  over  other 
information.  Jones  and  Endsley  (1996)  found  that  the  single  most  frequent  causal 
factor  associated  with  SA  errors  involved  situations  where  all  the  needed 
information  was  present,  but  was  not  attended  to  by  the  operator  (35%  of  total 
SA  errors).  This  was  most  often  associated  with  distraction  due  to  other  tasks. 
Correctly  prioritizing  information  in  a  dynamic  environment  remains  a  challenging 
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aspect  of  SA.  Good  SA  requires  enough  awareness  of  what  is  going  on  across  a 
wide  range  of  SA  requirements  (global  SA)  to  be  able  to  determine  where  to  best 
focus  one's  attention  for  more  detailed  information  (local  SA). 

4.  Mental  Workload 

Consideration  of  the  mental  workload  imposed  by  advanced  technology  is 
important  to  the  military  acquisition  community  and  technology  designers  who 
must  provide  systems  that  do  not  overburden  the  user  with  operating  tasks  and 
information  clutter.  Resource  or  capacity  models  of  mental  workload  postulate  a 
limited  quantity  of  resources  available  to  perform  a  task,  and  in  order  to  perform 
a  task,  one  must  use  some  or  all  of  these  resources.  These  models  of  workload 
address  the  difference  between  the  amount  of  resources  available  within  a 
person  and  the  amount  of  resources  demanded  by  the  task  situation  (McCloy, 
Derrick  and  Wickens,  1983).  Workload  also  refers  to  people’s  experience  of 
cognitive  task  performance  as  effortful  or  fatiguing,  which  may  index  task 
demands  and  attentional  overload  (Mulder,  1986).  Measures  of  workload  can  be 
classified  into  four  categories:  primary  task  measures,  secondary  task  measures, 
subjective  measures  and  physiological  measures.  In  this  thesis,  data  were 
collected  in  the  first  three  of  these  areas. 

5.  Human  Performance  -  Workload  Measurement 

We  defined  human  performance  as  a  measurable  outcome  of  a  certain 
task  conducted  by  humans.  Human  performance  is  complex  in  its  representation 
and  outcome.  In  many  tasks  it  is  difficult  to  describe  the  outcome  parameters  that 
are  expected.  For  instance,  an  attack  pilot  has  to  accomplish  a  wide  variety  of 
sub-tasks  in  order  to  fulfill  his  main  task:  attack  a  certain  object.  He  has  to 
operate  the  aircraft,  navigate  relative  to  the  ground  and  in  space,  to  coordinate 
his  actions  with  wingmen,  search,  and  select  the  target,  react  to  enemy  fire, 
maintain  radio  communication  with  several  sources,  select  the  appropriate 
weapon,  chose  the  right  moment  to  fire  the  weapon  etc.  Furthermore,  each  of 
these  tasks  consists  of  various  subtasks  (Wickens,  2002). 
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The  amount  of  information  processing  and  decision-making  required  in 
task  performance  impacts  the  workload  experienced  by  the  performer.  Hart  and 
Staveland  (1988)  add  to  this  definition  that  "Workload  is  not  an  inherent  property, 
but  rather  it  emerges  from  the  interaction  between  the  requirements  of  a  task,  the 
circumstances  under  which  it  is  performed,  and  the  skills,  behaviors,  and 
perceptions  of  the  operator." 

Two  major  ways  are  used  to  measure  human  performance:  objective 
measures  and  subjective  measures.  Objective  measures  are  concerned  with 
quantitative  and  qualitative  outcomes  of  human  performance.  Therefore  we 
measure  physical  variables  like  time  and  speed  or  we  measure  the  quality  of  the 
outcome  in  terms  of  error  rate  or  difference  to  the  desired  results.  A  subjective 
measurement  assesses  human  performance  through  questionnaires  or  self- 
reports,  which  maybe  considered  as  a  weaker  approach  (Endsley  et  al.,  2003). 
Objective  performance  based  measurement,  also  called  measures  of 
effectiveness  (MOEs)  or  measures  of  performance  (MOPs)  are  considered  as 
powerful  tools  for  measuring  the  performance  of  a  human-in-the-loop  system  and 
for  identifying  areas  of  inadequate  Situation  Awareness.  The  use  of  situations 
with  testable  responses  can  provide  valuable  insight  into  the  user's  Situation 
Awareness  and  how  the  user  will  act  upon  it  (Endsley  et  al.,  1998). 

Subjective  ratings,  are  the  scales  that  are  the  most  widely  used  workload 
assessment  tools.  The  advantages  of  subjective  workload  assessments  are  ease 
of  implementation,  low  cost,  and  limited  intrusion  on  task  performance.  They  are 
also  useful  in  evaluating  the  potential  for  task  overload  among  competing 
interface  designs.  They  can  be  applied  during  the  design  process  with  mock-ups, 
prototypes,  and  simulators,  as  well  as  assessing  workload  on  existing  systems. 
Two  subjective  rating  tools  frequently  used  in  aviation  and  other  safety-critical 
environments  are  the  NASA  Task  Load  Index  (NASA-TLX)  and  the  Subjective 
Workload  Assessment  Techniques  (SWAT)  which  have  subscales  assessing 
loads  for  time,  mental  effort,  and  psychological  stress.  NASA-TLX  allows  users  to 
perform  subjective  workload  assessments  on  operator(s)  working  with  various 
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human-machine  systems.  NASA-TLX  is  a  multi-dimensional  rating  procedure  that 
derives  an  overall  workload  score  based  on  a  weighted  average  of  ratings  on  six 
subscales.  These  subscales:  Mental  Demands,  Physical  Demands,  Temporal 
Demands,  Own  Performance,  Effort  and  Frustration.  It  can  be  used  to  assess 
workload  in  various  human-machine  environments  such  as  aircraft  cockpits; 
command,  control,  and  communication  (C3)  workstations;  supervisory  and 
process  control  environments;  simulations  and  laboratory  tests 
(www.nrl.navy.mil). 

Performance  Data  measures  whether  a  portion  or  a  user’s  entire  task  is 
selected  for  objective  and  quantitative  measurement.  Traditionally,  performance- 
based  measurements  have  included  speed  and  accuracy  data.  Speed  can  be 
reaction  time  either  to  perceive  an  event  or  to  initiate  a  response.  Accuracy  can 
be  defined  as  whether  or  not  a  signal  is  perceived  and  if  an  appropriate  response 
is  made.  Physiological  Measures  use  numbers  to  assess  a  cognitive  workload. 
These  measures,  in  general,  fall  into  one  of  two  classifications:  Background 
Measures  -  those  measures  not  linked  to  any  specific  ongoing  tasks  or  to  the 
timing  of  any  user  activity  or  response  and  Task-Related  -  those  measures 
specific  to  a  user  activity,  response,  or  an  event. 

Questionnaires  help  to  have  a  view  into  the  cognitive  model  of  the 
operator.  By  formulating  the  right  questions  the  researcher  is  able  to  gain  insight 
as  to  why  the  operator  acted  in  one  way  or  another.  It  helps  to  explain  “hard” 
data  of  the  objective  measurement  methods.  One  weakness  is  that 
questionnaires  are  subjective  (Matthews  et  al.,  2000),  greatly  depend  on  the  way 
questions  are  asked  and  the  experiment  is  organized  to  extract  the  desired 
information.  In  this  study  we  use  a  set  of  performance-based  measurement  of 
spatial  awareness  and  post-experiment  questionnaires  to  evaluate  subjective 
factors  to  better  explain  the  objective  measures  (Prevot  and  Palmer,  2000). 
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III.  PROPOSED  DISPLAY  DESIGN 


A.  GENERAL  BACKGROUND  AND  DESIGN  OF  WEBER  BOX 

1.  Background 

Navigation  awareness,  in  particular,  refers,  as  stated  in  previous  chapters, 
to  the  pilot's  dynamic  representation  or  mental  model  of  the  aircraft's  orientation 
within  two  dimensions,  space  and  time  (Endsley,  1988).  Flight  control  is 
characterized  by  three  spatial  coordinates  fx,  y,  and  z)  and  three  axes  of  rotation 
(pitch,  roll,  and  yaw),  providing  6  degrees  of  freedom  in  Euclidian  space. 
Conventional  display  formats,  however,  use  multiple  two-dimensional  (2-D) 
displays  to  provide  orthogonal  views  of  three-dimensional  (3-D)  flight  information. 
The  mental  model  is  formed  by  the  pilot's  perception  of  his  or  her  orientation  in 
the  outside  world  (e.g.,  attitude,  heading,  altitude,  airspeed),  which  is  based  on 
information  obtained  through  the  aircraft  displays,  the  forward  field  of  view, 
and/or  directly  from  the  pilot's  senses  (Endsley,  1988).  Several  human  factors 
arguments  could  be  made  for  the  implementation  of  3-D  technology  displays: 
whether  a  3-D  view  of  the  airspace  provides  a  more  "natural"  representation  than 
conventional  plan  view  (2-D)  displays,  or  if  a  single  3-D  perspective  display 
provides  a  more  "compatible"  view  by  reducing  the  need  to  integrate  mentally 
across  several  2-D  displays  (Woods,  1984).  According  to  the  initial  design  for 
Weber  Box,  pilots  would  benefit  with  information  displayed  to  them  that  would 
help  establish  and  monitor  SA,  enabling  them,  to  correctly  judge  the  proper 
attitude  at  any  given  time.  Thus,  Weber’s  (2006)  intention  was  to  represent  one 
display's  information  in  the  context  of  another  (Aretz,  1991)  in  order  to 
compensate  for  any  wrong  interpretation  of  the  information  provided.  Besides  the 
clear  representation,  a  mapping  of  the  real  world  is  provided  on  an  easy  to  learn 
set  of  instruments  which  creates  less  confusion  and  workload  for  the  operator 
(pilot  or  UAV  operator).  Demanding  less  attention,  the  operator  is  able  to  reduce 
reaction  time  for  tasks  associated  with  distraction  and  interruptions. 
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2.  Design  Goals  and  Purpose 

The  major  design  goal  is  to  support  the  operator’s  spatial  awareness  in  an 
easily  understandable  and  intuitive  way,  even  when  subjected  to  an  increased 
workload.  Traditional  flight  instrument  design  is  based  on  historic  design 
features,  which  are  very  abstract  and  non-intuitive  (Figure  9).  Conventional  flight 
instrument  displays  require  the  pilot  to  scan  instruments  looking  at  or  near  each 
of  a  number  of  instruments  in  succession  to  obtain  information.  Over  the  years 
many  different  layouts  have  been  experimented  with;  however  there  is  now  one 
accepted  layout,  which  all  modern  airplanes  adopt.  This  is  known  as  the  basic  T. 


ARTIFICIAL  HORIZON 


ALTIMETER 


AIRSPEED 


Figure  9.  Traditional  flight  instrument  design  (Screenshot  from  X-Plane  flight 

simulation) 

The  common  instrument  design  shown  in  Figure  9  demands  extensive 
training  to  develop  the  necessary  skills  to  interpret  and  use  these  displays 
(Fleadquarters  Department  of  the  Army,  1984).  In  extreme  situations,  fatal 
accidents  have  occurred  because  of  misinterpretation  of  the  cockpit  instruments 
(Roscoe,  2002). 
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Because  current  aircraft  display  design  inherits  many  poor  design  features 
from  its  historical  roots,  Weber  (2006)  decided  to  introduce  a  new  way  to 
represent  major  elements  of  flight  dynamics.  The  purpose  of  the  proposed  design 
was  to  introduce  an  innovative  way  to  display  spatial  information  on  flight 
instruments.  The  design  concept  therefore  breaks  with  traditional  flight 
instrument  design.  The  display  was  not  designed  to  substitute  any  of  the 
traditional  instruments.  It  rather  supports  the  operator  of  an  aircraft  in  extreme  or 
ambiguous  situations  by  intuitively  representing  an  appropriate  level  of  spatial 
awareness  (Weber,  2006). 

3.  Basic  Design  Concepts 

The  overall  display  layout  facilitates  the  implementation  into  HUDs  or 
HMDs.  Principles  of  abstract  and  simplified  symbology  and  restrictive  uses  of 
colors  are  applied  (Weber,  2006). 


Figure  10.  Screenshot:  overall  display  design  (From  Weber,  2006) 


The  texture  design  of  the  aircraft  was  aimed  to  support  definition  of  all 
possible  spatial  orientations.  The  top  side  has  a  green  wire  frame  on  black 
background;  the  under  side  has  a  black  wire  frame  on  green  background  (Figure 
12).  Furthermore,  the  top  side  texture  has  only  longitudinal  lines,  whereas  the 
bottom  side  has  squares.  The  application  receives  data  from  any  flight  simulator 
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via  network  communication  using  the  UDP  standard.  The  default  UDP  port  was 
49001  while  during  the  experiment  it  used  the  UDP  port  49002.  The  UDP  client 

TM 

demands  the  X-Plane  data  structure  specifications. 
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IV.  METHOD 


A.  METHODOLOGY 

1.  Experiment  Goals  and  Purpose 

The  goals  of  this  study  was  to  further  implement,  and  evaluate  the  Weber 
Box  proposed  display  design  (Weber,  2006)  that  supports  or  enhances  spatial 
awareness  in  3D  orientation  tasks  while  having  increased  workload  conditions.  In 
order  to  accomplish  this  goal  we  have  to  assess  the  impact  of  the  proposed 
display  design  on  people’s  spatial  awareness  in  an  environment  that  varies 
workload  experimentally. 

2.  Research  Questions 

Due  to  the  limited  scope  and  assets  of  this  study  this  evaluation  sought  to 
answer  four  main  questions:  (1)  How  will  the  display  design,  compared  to  a 
traditional  display  layout,  influence  the  time  to  assess  a  static  spatial  orientation? 
(2)  How  will  the  same  display  design  influence  the  time  to  assess  a  static  spatial 
orientation  while  operating  more  than  one  UAV  (3)  What  is  the  subjective 
impression  of  the  design  to  the  operator?  (4)  Will  the  impact  and  the  subjective 
judgment  be  different  for  participants  with  a  strong  aviation  background  versus 
participants  without  an  aviation  background  (pilots/non  pilots)? 

The  answer  to  the  first  two  questions  provides  insight  into  how  people  are 
able  to  build  their  mental  picture/model  of  the  current  spatial  situation  even  when 
they  have  to  operate  under  a  varying  workload  environment.  The  time  it  takes 
them  to  mentally  construct  their  spatial  awareness  enables  us  to  come  to 
conclusions  about  the  effectiveness  of  the  proposed  design  to  support  the  mental 
process  of  generating  spatial  awareness.  It  allows  us  at  least  estimate  the  impact 
of  the  display  on  real  world  tasks  and  enables  the  participants  to  evaluate  the 
design  in  the  context  of  a  goal-based  task. 

The  answer  of  the  third  question  gives  us  insight  about  the  impression  of 
the  participants  beyond  objective  performance  measurements.  It  enables  us  to 
bring  the  MOEs  in  context  and  may  help  to  diagnose  ambiguous  outcomes. 
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The  last  question  provides  information  about  the  relationship  between 
personal  background/training  and  the  user  acceptance  of  the  proposed  design. 
Perhaps  a  pilot  with  hundreds  or  thousands  of  hours  of  experience  with 
traditional  flight  instruments  will  find  it  much  easier  to  interpret  and  use  the  old 
instrumentation  and  harder  to  adapt  to  the  new  one  operating  under  an 
increasing  workload  environment  (multiples  UAVs).  The  opposite  might  be  the 
case  for  non-aviators. 

3.  Constraints  and  Assumptions 

Since  the  experiment  involves  human  participants  we  had  to  apply  for 
approval  by  the  Institutional  Review  Board  (IRB)  was  required.  The  experiment 
was  approved  as  proposed  (Appendix  A). 

The  flight  dynamics  were  provided  by  a  commercial  flight  simulation 
program,  X-Plane®  by  Laminar  Research™  in  version  7.13.  X-Plane  does  not 
allow  storing  complete  flight  situations,  including  flight  dynamics.  We  used  the 
original  cockpit  layout  generated  by  X-Plane,  and  a  model  of  a  Piper  PA-46 
Malibu,  which  is  a  small  single  propeller  turboprop  aircraft. 

4.  Technical  Equipment 

The  experiment  took  place  at  the  MOVES  Institute,  Naval  Postgraduate 
School  in  Monterey,  California.  The  proposed  display  was  implemented  on  a/two 
personal  computer  (Laptops)  in  addition  with  a/two  17-inch  TFT  monitor 
respectively,  which  was  placed  in  front  of  the  participant.  All  Laminar  Research 
software  uses  OpenGL  so  a  3-D  accelerator  card  that  can  run  OpenGL  was 
required  to  use  the  latest  software.  At  minimum  the  configuration  should  be  for 
Windows,  Pentium  1ghz+,  (or  for  Macintosh  PowerMac  800+),  with  disk  space  4 
gig,  a  3-D  card,  VRAM  at  least  32  meg,  and  a  monitor  at  a  1024x768+  resolution. 
A  peripheral  input  device  that  looks  similar  to  a  UAV  control  device,  a  computer 
joystick  (USB)  was  used  that  allowed  the  participants  to  control  two  UAVs  similar 
to  controlling  an  airplane  in  a  flight  simulator.  The  time  required  for  the  subject  to 
assess  spatial  attitude  for  both  sub-experiments  was  stopped  by  hand  using  a 
digital  hand  stopwatch  type  SPORTLINE®. 
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5.  Data  Collection  Methodology 

Both  sub-experiments  stopped  the  time  to  assess  the  current  spatial 
situation  by  hand  with  an  estimated  precision  of  ±0.1  seconds.  The  precision  of 
the  spatial  judgments  was  measured  by  evaluating  the  drawings  for  pitch  and  roll 
out  of  the  provided  schema  (Appendix  F).  The  drawings  were  evaluated  in  15° 
steps  based  on  the  error  from  the  given  values.  Hence,  the  precision  was  ±15°. 
Thus,  errors  within  15°  rounded  to  the  next  value. 

The  subjective  self-assessment  provided  a  scale  of  five  judgments  in 
every  questionnaire.  Demographic  data  were  collected  by  a  pre-experimental 
questionnaire  (Appendix  E).  Participants  completed  the  NASA-TLX  questionnaire 
immediately  following  each  of  the  two  sets. 

All  participants  were  exposed  to  all  instrument  setups  and  all  sub-experiments. 
The  order  of  presentation  of  the  two  of  displays  was  counterbalanced  using  a 
randomized  subject  assignment  procedure.  Selection  was  done  as  follows: 

•  List  participants  names 

•  Generate  random  numbers  of  enough  digits  that  each  exceeds  the 
size  of  the  sampling  list  by  several  digits.  This  makes  duplication 
unlikely. 

•  Assign  the  random  numbers  arbitrarily  to  individuals  in  the  sampling 
frame  list. 

•  Sort  the  list  of  random  numbers,  carrying  along  the  sampling  frame 
list. 

•  Now  the  first  n  values  in  the  sorted  sampling  frame  column  are  a 
simple  random  sample  (SRS)  of  n  values  from  the  entire  sampling 
frame. 

Odd-numbered  participants  started  with  the  traditional  instrument  setup; 
even-numbered  participants  started  with  the  new  instrument  setup.  This  ensured 
that  learning  effects  were  counter-balanced  over  the  experiment. 

6.  Data  Analysis  Methodology 

The  data  were  recorded  using  spreadsheets.  Every  sub-experiment  allows 
the  participant  six  individually  measured  and  recorded  trials.  The  data  were 
analyzed  by  commercial  statistical  software  JMP®  version  5.1.2,  using  various 
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methods  e.g.  linear  regression  analysis,  residual  analysis,  ANOVA  and  paired  t- 
tests.  Two  major  questionnaires,  subdivided  into  one  pre-experimental 
demographic  questionnaire,  and  two  post-experimental  subjective  evaluation 
questionnaires  where  administered  in  this  experiment  (Appendix  EandG).  The 
questionnaires  were  given  for  several  reasons:  as  to  develop  a  secondary 
method  to  determine  the  effects  of  the  dependant  variables,  and  to  enable  a 
correlation  between  a  participant’s  subjective  and  quantitative  data  and  to  assess 
the  relationship  between  subjective  ratings  of  the  workload  and  objective 
measures  of  performance  between  experienced  and  inexperienced  (novice) 
operators. 

B.  EXPERIMENT  DESIGN 

1.  Basic  Experiment  Design 

The  experiment  was  designed  to  compare  differences  in  time/speed  and 
accuracy/precision  while  changing  one  variable  (display  design),  while  tested  in 
different  workload  conditions  (two  UAVs).  The  dependent  variables  were  the 
outcomes  in  terms  of  differences  in  time  and  accuracy.  The  independent  variable 
was  the  instrument  layout  with  two  different  designs,  the  traditional  “basic  T”  and 
the  new  Weber  Box.  Participants  and  whether  operating  under  varying  workload 
conditions,  meaning  operating  one  or  two  UAVs  simultaneously. 

The  main  idea  was  to  have  a  set  of  two  sub-experiments  with  static  task  in 
order  to  measure  the  time  and  accuracy  of  the  participant’s  assessment  of  their 
spatial  situation.  It  also  served  the  purpose  of  making  the  participants  familiar 
with  the  use  and  interpretation  of  the  instruments  and  displays. 

The  participants  were  told  that  they  were  the  operator  of  a  combined  rotor-Zfixed- 
wing  based  Unmanned  Aerial  Vehicle  (UAV).  The  UAV  is  on  a  test  flight  and 
enters  a  sector  of  bad  weather.  The  data-link  connection  to  the  UAV  was 
interrupted  because  of  the  weather  conditions  (simulated  by  switching  off  the  17” 
monitor).  At  given  time  the  experimenter  freezes  the  simulation  software, 
providing  equally  difficult  attitudes  that  have  to  be  determined. 
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2.  Experimental  Procedure 

Every  participant  went  through  a  series  of  experiment  sets.  First  the 
participant  was  introduced  to  the  experiment.  Then  the  participant  filled  out  the 
necessary  paperwork,  including  consent  form,  privacy  statement,  and  minimal 
risk  information  form  (Appendix  A).  After  the  initial  brief,  the  participant  had  the 
opportunity  to  become  familiar  with  the  flight  simulator  and  control  devices  by  a 
free  self-paced  flight  training  while  assisted  by  the  experimenter. 

After  taking  the  pre-experimental  training  the  participant  started  with  task 
one  and  the  given  instrumentation  setup  (with  or  without  the  new  instrument  - 
Figure  11).  Subsequent  to  this  first  sub-experiment,  the  participant  starts  the 
second  sub-task,  while  this  time  he/she  had  to  operate  under  an  increased 
workload,  i.e.  controlling  two  UAVs.  Each  sub-experiment  set  was  scheduled  for 
20  minutes  with  an  optional  10  minute  break. 


EXPERIMENTER 


Figure  1 1 .  Instrumentation  setup  (task  dependant  respectively) 

When  the  participant  finished  the  second  experiment  block,  he/she  was 
asked  to  fill  out  the  post-experimental  questionnaire.  This  finished  the 
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experimental  session  for  this  subject.  The  detailed  experiment  is  laid  out  in  the 
experiment  protocols  (Appendix  B).  A  pilot  study  was  conducted  with  four 
volunteer  participants  to  test  procedures,  apparatus,  and  tasks  (having  the  same 
level  of  difficulty  in  defining  the  UAV’s  attitude). 

3.  Static  Spatial  Awareness  Experiment 

The  detailed  procedure  of  this  sub-experiment  was  documented  in  the 
experiment  protocol.  After  the  participant  was  led  into  the  lab,  he/she  was  shown 
all  the  equipment,  and  was  instructed  to  sit  in  the  operator  place.  The 
experimenter  explained  the  use  of  all  flight  instruments  and  controls  including  the 
Weber  Box,  which  was  referred  during  the  experiment  as  a  working  term  to  the 
proposed  display. 

The  participant  was  allowed  to  fly  for  about  5-10  minutes  as  a  brief 
training  to  become  familiar  with  controlling  the  UAV  where  explanations  for  all 
flight  instruments  and  controls  were  still  provided.  After  the  training  flight,  the 
experimenter  briefed  the  participant  for  the  first  task  (Recognition  of  spatial 
orientation  of  one  UAV,  Appendix  F).  Then,  the  instrumentation  monitor  or  the 
Weber  Box  respectively  was  switched  off,  depending  on  if  he  begins  with  or 
without  using  the  Weber  Box. 

The  first  trial  was  a  test  trial.  The  experimenter  picked  a  set  of  pitch-  and 
roll-angles  from  the  set  of  possible  angle  pairs  and  transferred  them  into  the  flight 
simulation  software.  When  the  experimenter  told  him/her  to  start,  while  switching 
on/off  respectively  the  monitors,  he/she  started  to  look  at  the  instruments.  When 
the  participant  said  “Stop!”  the  experimenter  measured  the  elapsed  time  and 
switched  off  the  monitor.  Then,  the  participant  had  to  draw  his/her  opinion  about 
pitch  and  roll  into  the  provided  schema  on  the  evaluation  sheet  (Appendix  F). 
The  experiment  started  over  with  the  next  experiment  test  and  a  new  set  of  pitch 
and  roll  angles  until  the  participant  finished  six  tasks  (plus  one  test  trial). 

4.  Static  Spatial  Awareness  Experiment  with  Increased  Workload 

The  detailed  procedure  of  this  sub-experiment  was  documented  in  the 

experiment  protocol  (Appendix  B).  As  the  participant  was  already  led  into  the  lab, 
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and  was  shown  all  equipment,  while  sitting  in  front  of  the  17”  monitor  he/she  had 
only  to  perform  the  exact  static  spatial  awareness  experiment  while  this  time 
he/she  was  asked  to  operate  two  UAV’s  (increased  workload). 

5.  NASA-TLX  over  Increased  Workload 

The  NASA-TLX  (Task  Load  index)  was  scored  for  each  participant  as 
follows.  First,  the  number  of  times  each  factor  was  circled  in  the  top  portion  was 
counted  to  determine  the  weight  for  that  factor.  The  total  number  of  weights 
summed  to  15.  Second,  a  scoring  template  was  made  with  the  subscales  divided 
into  20  equal  portions,  and  that  template  was  lined  up  behind  each  participant’s 
TLX  sheet.  The  numbered  portion  in  which  the  participant’s  mark  intersected  the 
line  determined  the  score  for  the  subscales.  In  cases  where  the  mark  appeared 
exactly  on  the  line  dividing  the  sections,  the  higher  number  was  given.  In 
addition,  in  cases  where  the  mark  did  not  touch  the  line,  a  mark  was 
extrapolated,  although  in  most  cases  the  majority  of  the  mark  was  in  the  section 
assigned.  In  cases  where  the  participant  had  written  a  checkmark  or  an  “x,”  the 
corner  of  the  check  or  intersection  of  the  lines  was  taken  as  the  mark,  and  the 
section  in  which  that  appeared  was  the  score  assigned.  Finally,  for  scales  in 
which  the  participant  had  not  marked  the  subscale,  the  mean  of  the  other  scores 
was  substituted.  For  each  factor,  the  weight  was  multiplied  by  the  score  on  the 
subscale  to  get  the  factor  score.  Finally,  the  factor  scores  were  added  together 
and  that  sum  was  divided  by  the  number  of  weights,  which  were  usually  15.  This 
number  was  the  total  workload  score. 

C.  PARTICIPANTS 

The  participants  were  randomly  assigned  from  the  twenty  students  of  the 
NPS  who  volunteered  for  the  study.  All  of  the  participants  were  male.  Nine 
participants  had  a  strong  aviation  background  (1  helicopter  pilot,  8  fighter-jet 
pilots).  The  mean  age  of  the  pilots  group  was  31  (range:  27  to  40)  and  the 
average  flight  experience  was  flight  hours  (range:  60  to  2500).  The  mean  age  of 
the  group  of  non-pilots  was  35  years  (range:  27  to  39). 
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V.  DATA  ANALYSIS 


A.  HYPOTHESES 

1.  Primary  Hypotheses 

The  primary  hypothesis  states  that  an  operator  will  be  able  to  assess 
spatial  position  significantly  faster  and  with  grater  accuracy  using  the  proposed 
instruments  (Weber  Box)  compared  to  use  of  the  traditional  instruments  when 
measured  under  increased  workload  conditions.  This  hypothesis  can  be  explored 
via  the  objective  results  of  the  static  spatial  awareness  experiment.  The 
measured  time  it  takes  a  participant  to  assess  a  particular  spatial  situation 
indicates  the  participant’s  ability  to  create  a  mental  model  of  the  current  spatial 
situation.  The  measured  angular  errors  (for  pitch  and  roll)  from  the  given  spatial 
orientation  indicated  the  accuracy  of  this  mental  model  (Weber,  2006). 

The  first  hypothesis  is  that  there  will  be  a  significant  difference  in  the  time 
required  to  assess  the  situation,  that  is,  delta  time=  time  (proposed  design)  - 
time  (traditional  design).  The  second  hypothesis  states  that  participants  this  time 
difference  also  will  be  true  in  the  increased  workload  condition.  We  derive  the 
null-hypothesis  that  there  is  no  proportional  difference  between  the  orientation 
times  of  the  two  designs  under  an  increased  workload.  Hence,  the  null- 
hypothesis  Ho  is:  =[//2-a]  3’’^^  jUjj=0,  and  therefore  Pi=  0,  that  is,  the  slope 

of  the  linear  regression  model  is  zero.  The  alternative  hypothesis  is  that  the 
orientation  time  using  the  proposed  design  is  proportionally  shorter  than  using 
the  traditional  design  under  increased  workload.  Hence,  Ha  is://2)<0>  srid 

therefore  Pi  <  0,  that  is,  the  slope  of  the  linear  regression  model  is  negative. 

2.  Secondary  Hypothesis 

As  mentioned  before,  the  measured  angular  errors  (for  pitch  and  roll)  from 
the  given  and  therefore  true  spatial  orientation  indicated  the  accuracy  of  this 
mental  model.  Thus,  our  secondary  hypothesis  was  that  the  angular  precision  of 
the  assessed  spatial  situation  was  at  least  at  the  same  level  for  both  instrument 
designs  under  increased  workload. 
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A  negative  sum  of  angular  errors  indicated  a  higher  accuracy  in  the  spatial 
awareness  assessment.  Equal  accuracy  indicates  that  the  new  design  was  at 
least  as  efficient  as  the  traditional  one,  even  while  operating  under  an  increased 
workload.  The  precision  was  measured  in  terms  of  angular  errors  in  15°  steps. 
This  means  that  errors  within  15°  were  rounded  to  the  next  value.  For  instance 
an  error  angle  of  17°  would  be  counted  as  a  15°  error  and  a  38°  error  would  be 
counted  as  a  45°  error  and  so  on. 

To  determine  the  number  of  participants  needed  in  the  study,  we  used 
statistical  software  package  JMP®  version  5.1.2.  In  this  Alpha  is  the  significance 
level,  in  this  study  0.05,  and  this  implies  willingness  to  accept  (if  the  true 
difference  between  groups  is  zero)  that  5%  (alpha)  of  the  time  a  significant 
difference  will  be  incorrectly  declared.  Error  Std  Deviation  is  the  true  residual 
error  and  Difference  to  detect  is  the  smallest  detectable  difference,  (how  small  a 
difference  you  want  to  be  able  to  declare  statistically  significant)  in  this  study  1 .4 
sec.  Sample  Size  is  the  total  number  of  observations  (runs,  experimental  units,  or 
samples)  and  Power  is  the  probability  of  getting  a  statistic  that  will  be  declared 
statistically  significant  in  this  study  0.85.  The  estimate  of  the  number  of 
participants  was  based  on  the  results  of  the  pilot  study. 
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^  JMP  -  [Sample  Size  and  Power] 
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Figure  12.  Statistics  of  required  sample  number 


B.  GENERAL  DATA  ANALYSIS 

1.  General  Data  Description 

The  data  analysis  was  supported  by  statistical  software  package  JMP® 
version  5.1.2  .  The  categories  were; 

•  Orientation  Time:  the  time  a  participant  needs  to  build  his/her 
spatial  awareness  in  seconds. 

•  Pitch  Error:  the  angular  error  of  the  estimated  pitch  in  degrees 

•  Roll  Error:  the  angular  error  of  the  estimated  roll  of  the  aircraft  in 

degrees. 

•  Demographic  questionnaire:  age  in  years  and  flight  experience  in 
hours. 

•  Post-Experimental  Questionnaire:  the  subjective  assessment  of  the 

experiment. 

2.  Demographic  Data 

The  non-pilot  participants  and  the  participant  with  aviation  background 

have  similar  means  of  the  age.  The  detailed  statistics  are  shown  in  Figure  14. 

41 


Mean 
Std  Dev 
Std  Err  Mean 
upper  95%  Mean 
lower  95%  Mean 
N 

Sum  Wgt 

Sum 

Variance 

Skewness 

Kurtosis 

CV 

N  Missing 


32.5 

4.5022003 

0.2005212 

33.19401 

32.40599 

480 

480 

15744 

19.300209 

-0.214051 

-0.530567 

13.393903 

0 


Figure  13.  Statistics  of  participant’s  age 
The  overall  statistics  for  the  age  are  as  follows: 


Mean 

StDev 

Minimum 

Maximum 

Age 

32.5 

4.502 

27 

40 

Pilots 

35 

3.887 

27 

39 

Non-Pilots 

31 

3.558 

27 

40 

Based  on  the  statistics  of  the  flight  hours,  we  were  able  to  conclude  that 
the  group  of  pilots  can  be  considered  as  experienced.  Most  of  the  pilot 
participants  had  more  than  1000  flight  hours;  only  three  of  the  nine  had  less  than 
1000  flight  hours  of  experience.  The  detailed  statistics  of  flight  experience  are 
shown  in  Figure  15. 
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Mean 

595.5 

Std  Dev 

902.98586 

Std  Err  Mean 

41.215477 

upper  95%  Mean 

676.48548 

lower  95%  Mean 

514.51452 

N 

480 

Sum  Wgt 

480 

Sum 

285840 

Variance 

815383.47 

Skewness 

1.1688686 

Kurtosis 

-0.285463 

CV 

151.63491 

N  Missing 

0 

Figure  14.  Statistics  of  participant’s  flight  experience 


3.  Orientation  Time  Effects 

The  comparison  of  the  means  and  the  distribution  graphs  does  not  reveal 
information  about  individual  differences  between  the  participants  but  based  on  a 
Tukey  -  Kramer  comparing  means  model  it  is  revealed  that  the  order  which  the 
participants  started  the  experiment,  meaning  starting  with  the  set  A  of  the 
traditional  instrument  or  set  B  of  Weber  Box,  basically  made  no  difference  in 
orientation  time  as  shown  in  Figure  16. 
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started  with  Box  (0/1 )  0.05 


Figure  15.  Linear  regression  on  Orientation  time  (all  participants) 

Also  we  can  state  that  there  were  no  differences  between  the  group  of 
pilots  and  non-pilots.  That  is,  pilots  were  slightly  faster  than  non-pilots  were  (0.45 
seconds).  Besides  the  practical  irrelevance  of  this  difference,  since  being  a  pilot 
was  a  non-significant  factor  in  the  overall  linear  regression  model,  this  difference 
has  no  statistical  relevance  as  shown  in  Figure  16. 
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Pilot?  (0/1)  0.05 


Figure  16.  Linear  regression  on  Orientation  time  (all  participants) 

When  we  compare  orientation  time  for  the  two  designs,  it  is  clearly 
observable  that  the  new  design  achieves  much  shorter  orientation  time.  In 
addition  we  were  able  to  determine  that  the  variance  of  the  data  decreased  by 
using  the  proposed  design. 

The  basic  evaluation  by  new  design  by  orientation  time,  with  a  level  of 
significance  of  a  =  0.01  by  providing  a  test  statistic  with  F  (1 ,478)  =  703.1474,  p  < 
0.0001  achieved  an  adjusted  of  0.5953,  which  states  that  this  model  explains 
60%  of  the  variances  in  the  data.  This  was  an  acceptable  value  to  establish 
statistical  significance.  Figure  18  shows  the  distance  between  the  two  sets  of 
orientation  time  for  the  linear  regression  model.  The  difference  in  the  means  of 
the  original  data  was  negative  7.95  seconds.  Thus,  it  took  an  operator  on 
average  the  3  times  longer  to  interpret  the  traditional  instruments. 
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Figure  17.  Linear  regression  on  Orientation  time  (all  participants) 


The  data  revealed  a  significant  difference  in  test  times  between  the  two 
set  ups  (operating  one  or  two  UAVs  simultaneously).  The  basic  evaluation  of 
workload  used  by  orientation  time,  with  a  level  of  significance  of  a  =  0.01  by 
providing  a  test  statistic  with  F  (1,478)  =  150.8302,  p  <  0.0001  we  achieved  an 
adjusted  of  0.2398,  which  states  that  this  model  explains  over  20%  of  the 
variances  in  the  data.  Figure  18  shows  the  distance  between  the  two  sets  of 
orientation  time  for  the  linear  regression  model. 

The  difference  in  the  means  of  the  original  data  was  5.05  seconds.  Thus, 
it  took  an  operator,  on  average,  2  times  longer  to  interpret  the  flight  instruments 
operating  under  increased  workload. 
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Tukey-Kramer 

Number  of  UAVs  (0/1 )  0.05 

Figure  18.  Linear  regression  on  Orientation  time  (all  participants) 

We  were  concerned  that  there  would  be  a  learning  effect  present  because 
participants  executed  6  tasks  per  set  up,  and  it  is  reasonable  to  expect  that  they 
might  acquire  proficiency  as  they  completed  more  tasks.  Figure  19  presents  an 
ANOVA  for  task  time  using  task  number  (1-6)  as  an  independent  variable.  Using 
a  t-test,  there  was  no  significant  difference  between  times  for  tasks  1  through  6. 
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Figure  19.  No  learning  effect  as  participants  progressed  through  the  tasks  for 

(all  participants) 


4.  General  Statistical  Methodology 

To  investigate  the  influences  of  the  designs  on  orientation  time,  we 
decided  to  conduct  a  multiple  regression  analysis.  First,  we  conducted  a 
stepwise  regression  analysis  (full  factorial)  to  eliminate  factors  that  were  not 
influential  in  our  model.  The  regression  model  that  we  are  developed  associates 
the  average  response  to  the  decision  factors.  We  began  by  using  only  main 
effects  to  get  an  idea  of  the  impact  of  these  factors  alone  on  our  model.  We 
developed  the  main-effects  model  using  the  mixed  stepwise  function  in  JMP. 
This  function  uses  alternating  forward  and  backward  steps,  allowing  terms  to 
enter  the  model  on  the  forward  step  and  leave  the  model  on  the  backward  step, 
based  on  a  significance  level  for  each.  For  our  model  we  allowed  terms  to  enter 
the  model  at  a  significance  of  .25  and  removed  terms  with  significance  less  than 
.01.  The  potential  predictors  were  workload  and  instrument  design  as  the  main 
suspected  factors,  and  aviation  experience  (pilot  or  not),  order  of  presentation 
and  participant’s  age.  The  participants  were  handled  as  a  blocking  factor  for  the 
regression  model.  The  response  variables  were  orientation  time,  and  pitch  and 
roll  error  time.  For  the  response  variable  orientation  time  only  the  display  design 

and  the  workload  had  significant  influence  as  a  predictor  variable  for  all  response 
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variables  with  F  (5,475)  =  1303.975,  p  <  0.000  and  F  (5,475)  =  652.783,  p  < 
0.000  respectfully.  Flowever,  the  graph  showed  that  the  display  design 
contributed  the  most  with  an  R^=0.83  while  the  workload  explained  the  data  with 
an  R^=0.25.  We  conducted  stepwise  regression  for  the  rest  of  the  response 
variables  with  similar  results. 


^  Stepwise  Fit 
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Figure  20.  Stepwise  regression  analysis  for  orientation  time 


The  regression  model  enables  us  to  reject  the  Ho  at  a  level  of  significance 
of  a  =  0.01  by  providing  a  test  statistic  with  F(1 5,464)  =  726.8835  (sufficiently 
large),  p  <  0.0001 ,  with  R^  of  0.9591 . 
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RSquare  0.959181 

RSquare  Adj  0.957861 

Root  Mean  Square  Error  1.060053 

Mean  of  Response  7.282292 

Observations  (or  Sum  Wgts)  480 


Source 

DF 

Sum  of  Squares 

Mean  Square 
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Model 
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816.808 

726.8835 

Error 

464 

521 .402 

1.124 

Prob  >  F 

C.  Total 

479 

12773.519 

<.0001 

Figure  21 .  Graph  explaining  test  time  as  a  result  of  the  independent  variables 


A  more  extensive  analysis  allowed  us  to  conduct  an  interpretation  with  the 
Prediction  profiler  (displays  prediction  traces  for  each  X  variable.  A  prediction 
trace  is  the  predicted  response  as  one  variable  is  changed  while  the  others  are 
held  constant  at  the  current  values)  in  JMP  where  an  interaction  plot  is  the 
evidence  of  interaction,  showed  by  nonparallel  lines.  The  importance  of  a  factor 
can  be  assessed  to  some  extent  by  the  steepness  of  the  prediction  trace.  If  there 
are  interaction  effects  or  cross-product  effects  in  the  model,  the  prediction  traces 
can  shift  their  slope  and  curvature  as  you  change  current  values  of  other  terms.  If 
there  are  no  interaction  effects,  the  traces  only  change  in  height,  not  slope  or 
shape.  We  can  notice  through  all  the  different  combinations,  that  among  the 
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following  idependent  variables  only  the  instrument  design  (new  design  (0/1))  and 
workload  (number  of  UAVs(0/1))  do  shift  the  slope  and  change  the  values  of  the 
terms,  with  the  first  having  the  most  influence. 


Pilot?  New  Design  Started  with  Number  of 

(0/1)  (0/1)  Box  (0/1)  UAVs  (0/1) 


Pilot?  New  Design  Started  with  Number  o 
(0/1)  (0/1)  Box  (0/1)  UAVs  (0/' 


Figure  22.  Prediction  profiler  analysis  for  orientation  time 

5.  Angular  Pitch  and  Roll  Errors  Effects 

The  statistics  of  the  pitch  and  roll  errors  show  that  participants  were  able 
to  judge  the  current  pitch  angle  of  the  UAV  more  accurately  using  the  new  design 
and  as  expected  they  performed  even  better  while  operating  under  an  increased 
workload  as  it  is  shown  in  the  Figure  23  and  24  (following).  The  box-plot  of  the 
data  suggests  a  significant  reduction  of  the  pitch  errors  by  the  new  design  which 
is  even  more  evident  while  operating  under  an  increased  workload. 

Although  the  data  revealed  that  there  was  no  significant  difference  in 
errors  between  the  two  set  ups  (operating  one  or  two  UAVs  simultaneously).  The 
difference  in  the  means  of  the  original  data  for  pitch  and  roll  by  design  (0/1)  was 
18.25°  and  01 .37°  and  06.25°  and  0.7  respectively.  Thus,  an  operator  on  average 
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made  the  same  proportional  errors  operating  the  new  design  under  an  increased 
workload  while  he  increased  the  errors  operating  with  the  traditional  instruments 
set  up. 


0  120  18.5000  1.9711 

1  120  36.7500  1.9711 


New  design 

Level  Number  Mean  Std  Error 

0  120  2.25000  0.82927 

1  120  3.62500  0.82927 


Figure  23.  Linear  regression  for  pitch  time  by  design  (0/1 ) 


Traditional  design  New  design 


Level 

Number 

Mean 

Std  Error 

Level 

Number 

Mean 

Std  Error 

0 

120 

76.3667 

4.0714 

0 

120 

6.37500 

1.5506 

1 

120 

84.1250 

4.0714 

1 

120 

7.00000 

1.5506 

Figure  24.  Linear  regression  for  roll  time  by  design  (0/1 ) 
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We  can  notice  through  all  the  different  combinations  of  independent 
variables,  only  the  instrument  design  (new  design  (0/1))  and  workload  (number  of 
UAVs(0/1))  do  shift  the  slope  and  change  the  values  of  the  terms,  with  instrument 
design  having  the  mostinfluence. 
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Figure  25. 
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Prediction  profiler  analysis  for  pitch  and  roll  time 
54 


The  distribution  of  the  angular  pitch  and  roll  error  can  be  assumed  as 
normal.  However,  the  box-plots  show  some  outliers,  which  we  decided  to  keep  in 
the  dataset.  These  outliers  represent,  in  our  opinion,  human  errors,  which  may 
occur  and  hence  were  not  an  abnormality.  In  particular  angular  error  can  lead  in 
many  cases  to  serious  aviation  mishaps  (Previc  et  al.,  2004). 

The  error  in  judging  their  own  roll  angle  was  mastered  by  pilots  and  non¬ 
pilots  with  an  equal  level  of  accuracy  and  a  similar  variance  of  the  data.  Pilots 
were  slightly  better  in  estimating  the  pitch  and  roll  of  the  UAV. 

6.  Post-Experiment  Questionnaires 

The  analysis  of  the  NASA-TLX  questionnaire  (of  the  subjective  workload 
of  all  the  operators  as  a  group)  showed  a  significantly  higher  workload  in  the 
increase  workload  setup.  When  analyzed  by  the  standard  of  aviation  experience, 
results  indicated  that  those  with  aviation  experience  reported  significant 
differences  in  workload  between  setups,  while  those  without  aviation  experience 
reported  more  significant  differences.  This  indicates  that  the  operators’  subjective 
workload  of  operators’  increases  with  the  addition  of  secondary  tasks,  but  this 
may  vary  with  aviation  experience.  The  mean  NASA-TLX  scores  are  presented  in 
Table  1. 


Table  1 .  Mean  Subjective  Workload  Scores 


1  UAV  TLX  score 

2  UAVs  TLX  score 

All  participants 

62.88 

65.48 

Pilots 

58.24 

61.38 

Non  pilots 

64.82 

66.53 

The  difficulty  of  the  tasks  was  experienced  as  slightly  difficult  to  somewhat 
easy  to  accomplish.  The  answers  of  the  post-experimental  questionnaires 
showed  a  strong  tendency  towards  the  preference  of  the  Weber  Box.  However, 
the  questionnaires  show  that  most  participants  (over  80%),  pilots  and  non-pilots. 
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appreciated  the  proposed  design  and  felt  supported  in  their  spatial  orientation 
tasks  more  than  using  the  traditional  setup  (basic  T).  The  answers  in 
percentages  were  as  follows. 


Table  2.  Results  of  Post-Experiment  Questionnaires  -  Orientation  Task  A 


Question 

Answer-Options 

In  general,  how  do  you  judge  the 
difficulty  of  evaluating  the  current 
attitude  of  the  UAV? 

Very 

easy 

5% 

Somewhat 

easy 

40%) 

Border 

line 

20% 

Somewhat 

complicated 

35% 

Very  difficult 

0% 

In  general,  how  do  you  judge 
difficulty  of  using  the  traditional  flight 
instruments  for  this  task? 

Very 

easy 

0% 

Somewhat 

easy 

20% 

Border 

line 

20% 

Somewhat 

complicated 

50% 

Very  difficult 

15% 

In  general,  how  do  you  judge  the 
difficulty  of  using  the  WEBER-Box  for 
this  task? 

Very 

easy 

50% 

Somewhat 

easy 

50% 

Border 

line 

0% 

Somewhat 

complicated 

0% 

Very  difficult 

0% 

Did  you  feel  better  aware  of  the 
spatial  orientation  of  the  UAV  with 
the  WEBER-Box  in  comparison  with 
the  traditional  flight  instruments? 

Much  less 

0% 

Somewhat 

less 

0% 

Border 

line 

0% 

Somewhat 

more 

25% 

Much  more 

75% 

Did  having  the  WEBER-Box  increase 
or  decrease  your  monitoring 
demands  in  comparison  with  the 
traditional  flight  instruments? 

Greatly 

decreased 

45% 

Somewhat 

decreased 

50% 

Unaffected 

5% 

Somewhat 

Increased 

0% 

Greatly 

Increased 

0% 

Did  having  the  WEBER-Box  increase 
or  decrease  your  overall  workload  in 
comparison  with  the  traditional  flight 
instruments? 

Greatly 

decreased 

45% 

Somewhat 

decreased 

50% 

Unaffected 

5% 

Somewhat 

Increased 

0% 

Greatly 

Increased 

0% 

Did  having  the  WEBER-Box  increase 
or  decrease  your  overall  frustration 
level  (irritate,  stress,  insecure)  in 
comparison  with  the  traditional  flight 
instruments? 

Greatly 

decreased 

70% 

Somewhat 

decreased 

25% 

Unaffected 

5% 

Somewhat 

Increased 

0% 

Greatly 

increased 

0% 

How  would  you  characterize  your 
Computer  Skills 

None 

0% 

Fair 

0% 

Average 

10% 

Good 

70% 

Excellent 

20% 

How  much  experience  do  you  have 
with  computer  games? 

None 

Fair 

Average 

30% 

Good 

60% 

Excellent 

10% 
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VI.  CONCLUSIONS 


A.  OVERALL  ASSESSMENT  OF  THE  STATISTICAL  RESULTS 

The  purpose  of  the  current  study  was  to  assess  both  subjective  workload 
and  associated  performance  decrements  while  operating  the  proposed  display 
design  versus  the  traditional  instrument  design  (Weber,  2006).  Specifically,  the 
focus  was  to  assess  the  relationship  between  a  subjective  workload  and  various 
objective  measures  of  performance  both  with  and  without  the  introduction  of  a 
secondary  task  involving  operating  one  or  two  UAVs. 

The  statistical  analysis  supported  the  proposed  design  in  all  aspects.  The 
design  seemed  to  sustain  remarkably  well  the  spatial  awareness  in  3D 
orientation  tasks  even  under  high  workload  conditions.  Time  to  assess  a  spatial 
situation  decreases  significantly,  with  the  new  display,  whereas  accuracy  of  this 
spatial  judgment  at  least  maintains  its  level.  Judgment  errors  were  minimized  and 
the  extreme  errors  were  almost  eliminated.  Figure  26  shows  the  level  of 
increased  orientation  time  as  workload  increases  while  operating  the  traditional 
design.  The  steeper  the  line  shows,  the  more  influence  is  of  the  added  workload. 
It  seems  that  the  pilots  were  not  able  to  improve  their  speed  better  than  non¬ 
pilots  but  further  analysis  revealed  that  was  due  to  increased  accuracy. 


Figure  26.  Linear  regression  for  orientation  time  by  design  (traditional) 
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Figure  27  shows  the  level  of  increased  orientation  time  as  the  workload 
increases  while  operating  the  proposed  design.  The  added  workload  is  more 
influential  when  the  line  is  steeper.  It  seems  that  the  pilots  were  able  to  improve 
their  speed  better  than  non-pilots  did. 


Figure  27.  Linear  regression  for  orientation  time  by  design  (proposed) 


It  was  surprising  that  pilots  and  non-pilots  were  so  similar.  The  differences 
were  minor.  We  explain  this  phenomenon  with  the  character  of  the  tests,  which 
were  not  overly  demanding.  Another  aspect  might  be  that  the  underlying 
cognitive  processes  are  basically  the  same  whether  operating  the  Weber  Box 
design  (while  it  was  almost  twice  as  big)  or  operating  the  setup  with  the 
traditional  instruments.  Thus,  everybody  took  equal  advantage  out  of  the 
proposed  design. 


The  statistics  of  the  values  were  as  follows: 


Traditional  design 

Source 

DF 

F  Ratio 

Prob  >  F 

Number  of  UAVs  1 

1655.561 

<.0001 

(0/1) 

Error 

238 

C.  Total 

239 

Level  Number 

Mean 

Std  Error 

Lower  95% 

Upper  95% 

0  120 

7.0550 

0.14624 

6.767 

7.343 

1  120 

15.4700 

0.14624 

15.182 

15.758 
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Proposed  design 

Source 

DF 

F  Ratio 

Prob  >  F 

Number  of  UAVs  (0/1 ) 

1 

295.2498  <.0001 

Error 

238 

C.  Total 

Level  Number  Mean 

239 

Std  Error 

Lower  95% 

Upper  95% 

0  120  2.45667 

0.06958 

2.3196 

2.5937 

1  120  4.14750 

0.06958 

4.0104 

4.2846 

The  proposed  study  was  conducted  to  support  operators  of  UAVs  in 
extreme  situations  -  increased  workload.  The  main  goal  was  to  provide  an 
intuitive  way  to  improve  the  level  of  spatial  awareness  in  ambiguous  conditions. 
In  particular,  it  utilizes  the  principle  of  the  mental  model  by  introducing  an 
exocentric  view  on  a  virtual  avatar  in  an  abstract  three-dimensional  virtual  world 
while  the  participants  were  subjected  to  increased  workload.  This  positive  picture 
was  supported  by  the  comments  and  post-experimental  questionnaires.  The 
majority  of  the  participants  found  the  “WEBER-Box”  to  be  very  helpful  and 
intuitive  to  understand,  especially  under  increased  workload  where  the  response 
time  was  proportionally  smaller.  The  participants  felt  that  the  “WEBER-Box” 
improves  their  spatial  awareness  and  made  it  easier  to  accomplish  their  tasks. 
The  following  Figure  28  depicts  JMP’s  partition  platform  which  enables  us  to 
systematically  analyze  data  sets  and  to  discover  unsuspected  or  unknown 
relationships  by  visualizing  a  successive  tree  of  partitions  according  to  a 
relationship  between  the  X  and  Y  variables.  It  finds  a  set  of  cuts  or  groupings  of 
X  values  that  best  predict  a  Y  value  by  exhaustively  searching  all  possible  cuts  or 
groupings,  recursively.  Thus  we  can  easily  see,  achieving  R^  of  0.959,  the  impact 
of  the  design  as  opposed  to  the  increased  workload.  Similar  partition  model  were 
found  for  roll  and  pitch  errors  indicating  the  superiority  of  the  Weber  box. 


59 


Figure  28.  Partition  for  orientation  time 

Many  of  the  underlying  interactions  and  effects  are  not  completely 
investigated.  Questions  about  design  improvements  and  practical  applications 
need  to  be  answered  as  well.  However,  by  applying  human-centered  design 
principles  we  were  able  to  design  an  efficient  tool  to  support  spatial  awareness  in 
3D-orientataion  tasks.  Our  data  clearly  reveals  that  the  nontraditional  design 
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Weber  box  (Weber,  2006)  yielded  superior  performance  to  the  traditional  one. 
The  design  was  accepted  and  appreciated  by  both  the  pilot  and  the  non-pilot 
participants  and  was  proven  beneficial  especially  under  complex  tasks  with 
increased  workload. 

B.  FUTURE  APPLICATIONS 

We  can  see  applications  of  the  WEBER  Box  design  in  all  major  fields  of 
aviation.  Since  this  design  is  intuitively  understandable,  it  might  help  flight 
students  to  understand  traditional  flight  instruments.  Thus,  integration  into 
traditional  flight  instruments  or  even  to  add  flight  instruments  to  WEBER  Box 
might  be  beneficial.  Because  this  study’s  major  goal  was  to  implement,  test  and 
evaluate  a  novel  display  design  (Weber,  2006),  it  turned  out  that  more  questions 
were  formed  than  it  answered.  Foremost  questions  that  are  open  to  future 
research  are  about  design  improvements  and  the  influence  of  alternations  and 
simplifications.  Questions  may  include: 

•  What  is  the  optimal  size? 

•  How  much  can  we  simplify  the  aircraft  model? 

•  Should  we  integrate  scales  to  make  it  a  complete  flight  instrument 
and  given  that,  how  much  information  and  graphics  can  be 
simplified  without  degrading  the  outcome? 

•  Are  the  principles  of  the  WEBER  Box  applicable  to  other  fields  (car 
driving,  underwater  robots,  etc.)? 

•  Is  the  WEBER  Box  useful  for  training  purposes  as  pilot  training, 
UAV  ground  control  systems  operators,  etc.? 

•  How  much  an  even  more  increased  workload  environment  (i.e. 
auditory  tasks)  will  affect  performance  and  spatial  attitude 
definition? 

Other  research  might  investigate  the  potential  of  the  current  design,  the 
regarding  the  altitude  representation  and  the  various  possible  implementations  of 
warning  at  extreme  situations. 
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APPENDIX  A.  IRB  APPLICATION  AND  APPROVAL 


Anthony  Ciavarelli,  Ed.D. 

Modeling  Virtual  Environment  and  Simulation 
Department 
Watkins  Hall 

Naval  Postgraduate  School 
Monterey,  California  93943 


831-656-2191 
DSN:  756-2191 
Fax:  831-656-2595 
ACiavarelli@nps.edu 


To:  Protection  of  Human  Subjects  Committee 

Subject:  Application  for  Human  Subjects  Review  for  Use  of  Avatars  to 
Support  and  Enhance  Spatial  Awareness  in  3D-Orientation-Tasks 


1.  Attached  is  a  set  of  documents  outlining  a  proposed  experiment  to  be 
conducted  over  the  year  to  support  the  thesis  of  Lieutenant  Colonel  Dimitries 
Myttas,  GR. 

2.  We  are  requesting  approval  of  the  described  experimental  protocol.  An 
experimental  outline  is  included  for  your  reference  that  describes  the  methods 
and  measures  we  plan  to  use. 

3.  We  include  the  consent  forms,  privacy  act  statements,  questionnaires,  and 
briefing  forms  we  will  be  using  in  the  experiment. 

4.  We  understand  that  any  modifications  to  the  protocol  or 
instruments/measures  will  require  submission  of  updated  IRB  paperwork  and 
possible  re-review.  Similarly,  we  understand  that  any  untoward  event  or  injury 
that  involves  a  research  participant  will  be  reported  immediately  to  the  IRB 
Chair  and  NPS  Dean  of  Research. 


Anthony  Ciavarelli 
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APPLICATION  FOR 
HUMAN  SUBJECTS  REVIEW 


HSR  NUMBER 


(HSR) 


(to  be  assigned) 


PRINCIPAL  INVESTIGATOR(S)  (Full  Name,  Code,  Telephone) 

Anthony  Ciavarelli,  Ed.D.,  Monterey,  CA  93943,  Phone:  831  656-2191 
(Thesis  LtCol  Dimitrios  Myttas,  GR) 


APPROVAL  REQUESTED  [  X  ]  New  [  ]  Renewal 


LEVEL  OF  RISK  [  ]  Exempt  [  ]  Minimal  [  ]  More  than  Minimal 
Justification:  3D-Virtual  Environment  (flight  simulator  software) 


WORK  WILL  BE  DONE  IN  (Site/Bldg/Rm) 

NPS,  Watkins  Hall,  Rm  212B 

ESTIMATED  NUMBER  OF  DAYS  TO 
COMPLETE 

10 

MAXIMUM  NUMBER  OF  SUBJECTS 

30 

ESTIMATED  LENGTH  OF  EACH 
SUBJECT’S  PARTICIPATION 

2  hours 

SPECIAL  POPULATIONS  THAT  WILL  BE  USED  AS  SUBJECTS 

[]  Subordinates  []  Minors  [  X  ]  NPS  Students  []  Special  Needs  (e.g.  Pregnant 
women) 

Specify  safeguards  to  avoid  undue  influence  and  protect  subject’s  rights:  none 


OUTSIDE  COOPERATING  INVESTIGATORS  AND  AGENCIES 


-  none- 

[  ]  A  copy  of  the  cooperating  institution’s  HSR  decision  is  attached. _ 

TITLE  OF  EXPERIMENT  AND  DESCRIPTION  OF  RESEARCH  (attach  additional 
sheet  if  needed).  Methodology  attached 


I  have  read  and  understand  NPS  Notice  on  the  Protection  of  Human  Subjects.  If  there 
are  any  changes  in  any  of  the  above  information  or  any  changes  to  the  attached 
Protocol,  Consent  Form,  or  Debriefing  Statement,  I  will  suspend  the  experiment  until  I 
obtain  new  Committee  approval. 

SIGNATURE _ 

DATE 
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APPENDIX  B.  EXPERIMENT  PROTOCOL 


EXPERIMENTAL  WORKSTATION  ENVIRONMENT  STARTUP 


1. 

2. 

3. 

4. 

5. 


6. 


Switch  on  the  personal  computer  (laptop),  and  the  extra  17”  TFT  monitor 
on  the  experimenter  desk. 

Login  at  the  computer  by  pressing  ALT_DEL_CTRL.  Let  the  password 
empty  and  press  ENTER. 

Open  the  “Local  Area  Network”  icon  in  the  task  bar  using  the  RIGHT 
mouse  button. 

In  the  following  menu  chose  STATUS,  click  on  the  SUPPORT-tab  and 
write  down  the  IP-address  (e.g.  131.120.151.6),  or  just  type  “ipconfig”  at 
command  prompt. 

Start  on  the  desktop  the  following  programs  by  clicking  the  icons  twice  by 
the  mouse: 

a.  “UDP  Reflector”  =>  type  in  the 
dialog  box  the  IP-address  of  the 
computer  und  press  ENTER 

b.  “X-Plane  Experiment” 

After  “X-Plane”  started. 


IP  Address  of  2nd  Computer: 
131.120.XXX.XXX 


1)1 27.0.0.1 


choose  menu  “Location”  =>  select  “Place 
Aircraft  by  Airport”  =>  chose  “Twenty-nine 
Palms  EAF”  =>  click  “Go  to  this  Airport” 
choose  menu  “Files”  =>  select  “Load 
Scenarion”  =>  click  on  “Experiment.sit” 
wait  until  the  situation  is  loaded,  than  hit  the 
“P”  button  of  the  keyboard  to  pause  the 
simulation 

make  sure  that  the  following  values  have 
been  typed  (127.0.0.1  -  localhost). 


Start  the  WEBER  BOX 

1 .  On  the  computer  double  click  on  the  icon  ’’Start  Weber  Box”  icon  ->  in 
the  window  type  in  for  the  UDP  port:  49002  and  1.0  for  the  speed.  Now  the 
WEBER-Box  should  appear.  The  experimenter  can  drag  it  to  the  17”  monitor 
for  all  experiments  which  demand  the  WEBER-Box. 

2.  Double-click  on  the  icon  “Experimenter”  to  start  the  input  consol. 

Type  in  the  field  “IP-Address”  the  number 
131.120.150.248  and  click  the  button  “set” 
right  to  the  input  field 

Type  in  the  filed  “ID”  the  number  16  and 
click  the  button  “int”  below  the  field 
=>  Insert  the  different  values  (VO,  VI-  Pitch, 
Roll)  and  click  the  button  “fit”  below  the  field 
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PRE-EXPERIMENT  PROTOCOL 

Check  if  all  forms  are  prepared  and  signed  by  the  experimenter  and  the 

principle  investigator 

Escort  participant  to  peroration  desk 

Administer  Initial  Questionnaires 

o  Hand  out  the  written  experiment  objectives  and  the  experiment 
introduction. 

o  Present  Institutional  “IRB  Participant  Consent  Form”,  the  “Minimal 
Risk  Consent  Statement”,  the  “Privacy  Act  Statement”  and  let  it 
sign  by  the  participant. 

o  Check  if  the  participant’s  ID  number  is  noted  on  every  page  of  the 
initial  questionnaire  and  present  it  and  let  the  participant  sign  it. 
Request  if  the  participant  has  any  further  questions 

SPATIAL  AWARENESS  EXPERIMENT  PROTOCOL 

Participant  is  led  to  the  Experimental  workstation  and  shown  all 
equipment. 

Participant  is  instructed  to  sit  in  front  of  the  operator  computer. 
Experimenter  explains  all  flight  instruments  and  controls  incl.  the  Weber 
Box. 

Experimenter  starts  the  training  scenario  (the  aircraft  is  airborne  on  a  safe 
altitude  and  attitude)  and  allows  the  participant  to  fly  for  about  10  minutes. 
In  this  time  the  experimenter  provides  any  help  regarding  explanations  of 
the  use  of  the  flight  instruments  and  controls.  He  must  not  refer  to  any  of 
the  later  experiment  tasks. 

Experimenter  gives  the  participant  the  briefing  for  the  first  task 
(Recognition  of  spatial  orientation  of  the  aircraft)  and  makes  participant 
familiar  with  the  procedures.  He  is  now  advised  to  read  the  task 
instructions  for  this  task. 

Experimenter  switches  off  the  instrumentation  monitor  or  switches  off  the 
Weber  Box  respectively,  depending  if  he  begins  with  or  without  using  the 
Weber  Box. 

Experimenter  asks  participant  if  he/she  understands  everything  and  if  he 
may  start  the  experiment 

The  first  trial  is  a  test  trial;  the  test-trial  sheet  has  to  be  used  for  it. 
Experimenter: 

o  starts  spatial  awareness  scenario 
o  prepares  the  stop  watch 

Experimenter:  Switches  off  the  monitor,  and  determines  a  set  of  pitch-  and 
roll-angles  from  the  task  sheet  “Test  A”  (“Test  B”  for  the  2"^^  set  of 
experiments)  by 

o  Typing  the  angle  values  in  degrees  in  field  “00”  for  the  pitch  and 
field  “01”  for  the  roll  and  clicks  on  the  button  “fit”  below  the  field  to 
store  the  values 


o  Clicks  the  button  “Send  Data  Package”  to  transfer  the  values  to  the 
experimental  workstation  monitor 

o  Observes  whether  the  flight  simulation  software  (X-Plane  or  Weber 
Box)  reacts  as  expected 

o  If  not,  he  checks  again  all  values  and  presses  all  described  buttons 
again  to  store  the  data 

•  Experimenter: 

o  Stands  directly  on  the  participants  left  side  and  places  his  left  hand 
at  the  power  button  of  the  monitor  and  holds  the  stop  watch  in  his 
right  hand 

o  Switches  on  the  monitor  and  tells  the  participant  that  he  may  start 
to  look  at  the  instruments 

o  In  the  moment  the  participant  starts  looking  the  monitor  and  at  the 
instruments,  he  starts  the  stop  watch 
o  When  the  participant  says  “Stop!”  he  switches  of  the  monitor  and 
stops  the  stop  watch  at  the  same  time 

•  Experimenter: 

o  Let  the  participant  draw  his/her  opinion  about  pitch  and  roll  into  the 
provided  schema  on  the  evaluation  sheet 
o  Writes  the  measured  time  into  field  next  to  the  evaluation  schema, 
o  Experimenter  starts  over  with  the  next  experiment  loop  and  a  new 
set  of  pitch  and  roll  angles  until  the  participant  finished  six  tasks 
(plus  one  test  trial). 

•  Experimenter  starts  with  the  SPATIAL  AWARENESS  EXPERIMENT  with 
increased  workload  (Operating  two  UAVs) 

POST  EXPERIMENT  PROTOCOL 

•  Participant  is  allowed  to  have  a  five  minutes  break 

•  Reseat  the  participant  and  start  with  the  2"'^  block  of  experiments  including 
the  spatial  awareness  experiment  and  spatial  awareness  experiment  with 
increased  workload. 

•  After  all  tests  are  completed  the  experimenter: 

o  Thanks  the  participant  for  his/her  effort 
o  Guides  participant  to  a  chair 

o  Provides  the  “Post-Experiment  Questionnaire”  and  asks  participant 
to  fill  it  out 

o  Evaluates  the  angular  errors  of  the  SA-experiment  and  notes  it 
down  into  the  evaluation  sheets 
o  Records  all  results  in  the  participant’s  database. 

•  Thank  participant  for  participating  in  the  experiment. 
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APPENDIX  C.  PARTICIPANT  INFORMATION  FORMS 


PARTICIPANT  CONSENT  FORM 

1 .  Introduction.  You  are  invited  to  participate  in  a  study  of  spatial  awareness  of  virtual 
environments.  With  information  gathered  from  you  and  other  participants,  we  hope  to 
prove  the  concept  of  the  use  of  virtual  avatars  to  support  and  enhance  spatial 
awareness  and  improve  orientation  performance.  We  ask  you  to  read  and  sign  this 
form  indicating  that  you  agree  to  be  in  the  study.  Please  ask  any  questions  you  may 
have  before  signing. 

2.  Background  Information.  The  Naval  Postgraduate  School  MOVES  Institute  is 
conducting  this  study. 

3.  Procedures.  If  you  agree  to  participate  in  this  study,  the  researcher  will  explain  the 
tasks  in  detail.  There  will  be  two  sessions  with  two  different  orientation  tasks  each. 
The  execution  phases  will  last  approximately  two  hours  total,  during  which  you  will  be 
asked  to  accomplish  a  number  of  tasks  related  to  spatial  awareness  while  operating  a 
virtual  flight  simulator  software.  Following  the  study  you  will  be  asked  to  fill  out 
questionnaires  during  a  20-minutes  rest  period. 

4.  Risks  and  Benefits.  This  research  involves  having  a  participant  fly  a  3D  virtual 
flight  simulator  software.  If  you  have  any  cardiac  risk  factors  (High  blood  pressure, 
smoking,  diabetes,  high  cholesterol,  previous  heart  problems),  we  request  that  you 
PLEASE  INFORM  THE  EXPERIMENT  ADMINISTRATOR  AND  YOU  WILL  NOT  BE 
ABLE  PARTICIPATE  IN  THE  EXPERIMENT. 

5.  Compensation.  No  tangible  reward  will  be  given.  A  copy  of  the  results  will  be 
available  to  you  at  the  conclusion  of  the  experiment  period. 

6.  Confidentiality.  The  records  of  this  study  will  be  kept  confidential.  No  information  will 
be  publicly  accessible  which  could  identify  you  as  a  participant. 

7.  Voluntary  Nature  of  the  Study.  If  you  agree  to  participate,  you  are  free  to  withdraw 
from  the  study  at  any  time  without  prejudice.  You  will  be  provided  a  copy  of  this  form 
for  your  records. 

8.  Points  of  Contact.  If  you  have  any  further  questions  or  comments  after  the 
completion  of  the  study,  you  may  contact  the  research  supervisor.  Dr.  Anthony 
Ciavarelli,  Ed.D.,  NPS  Monterey,  CA  93943,  Phone:  831  656-1073 

9.  Statement  of  Consent.  I  have  read  the  above  information.  I  have  asked  all  questions 
and  have  had  my  questions  answered.  I  agree  to  participate  in  this  study. 


Participant’s  Signature 


Date 


Researcher’s  Signature 


Date 
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MINIMAL  RISK  CONSENT  STATEMENT 

NAVAL  POSTGRADUATE  SCHOOL,  MONTEREY,  CA  93943 

Participant:  VOLUNTARY  CONSENT  TO  BE  A  RESEARCH  PARTICIPANT  IN: 

Use  of  Avatars  to  Support  and  Enhance  Spatial  Awareness  in  3D- 

Orientation-Tasks 


1.  I  have  read,  understand  and  been  provided  "Information  for  Participants"  that 
provides  the  details  of  the  acknowledgments  below. 

2.  I  understand  that  this  project  involves  research.  An  explanation  of  the  purposes  of 
the  research,  a  description  of  procedures  to  be  used,  identification  of  experimental 
procedures,  and  the  extended  duration  of  my  participation  have  been  provided  to 
me. 

3.  I  understand  that  this  project  does  not  involve  more  than  minimal  risk.  I  have  been 
informed  of  any  reasonably  foreseeable  risks  or  discomforts  to  me. 

4.  I  have  been  informed  of  any  benefits  to  me  or  to  others  that  may  reasonably  be 
expected  from  the  research. 

5.  I  have  signed  a  statement  describing  the  extent  to  which  confidentiality  of  records 
identifying  me  will  be  maintained. 

6.  I  have  been  informed  of  any  compensation  and/or  medical  treatments  available  if 
injury  occurs  and  is  so,  what  they  consist  of,  or  where  further  information  may  be 
obtained. 

7.  I  understand  that  my  participation  in  this  project  is  voluntary;  refusal  to  participate  will 
involve  no  penalty  or  loss  of  benefits  to  which  I  am  otherwise  entitled.  I  also 
understand  that  I  may  discontinue  participation  at  any  time  without  penalty  or  loss  of 
benefits  to  which  I  am  otherwise  entitled. 

8.  I  understand  that  the  individual  to  contact  should  I  need  answers  to  pertinent 

questions  about  the  research  is  Professor  Anthony  Ciavarelli,  Thesis  Advisor,  and 
about  my  rights  as  a  research  participant  or  concerning  a  research  related  injury  is 
Prof.  Rudy  Darken,  MOVES  Institute  Chairman  or  the  NPS  IRB  Medical  Advisor, 
LTC  Eric  Morgan,  MC,  USA,  Presidio  of  Monterey,  (831)  242-7550, 

eric.morgan@NW.AMEDD.ARMY.MIL 


Signature  of  Principal  Investigator  Date 


Signature  of  Volunteer 


Date 
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PRIVACY  ACT  STATEMENT 


NAVAL  POSTGRADUATE  SCHOOL,  MONTEREY,  CA  93943 
PRIVACY  ACT  STATEMENT 

1.  Purpose;  Spatial  cognition  data  will  be  collected  to  enhance  knowledge,  and 
to  develop  tests,  procedures,  and  equipment  to  improve  the  development  of 
Virtual  Environments. 

2.  Use:  Spatial  cognition  data  will  be  used  for  statistical  analysis  by  the 
Departments  of  the  Navy  and  Defense,  and  other  U.S.  Government 
agencies,  provided  this  use  is  compatible  with  the  purpose  for  which  the 
information  was  collected.  Use  of  the  information  may  be  granted  to 
legitimate  non-government  agencies  or  individuals  by  the  Naval 
Postgraduate  School  in  accordance  with  the  provisions  of  the  Freedom  of 
Information  Act. 

3.  Disclosure/Confidentiality: 

a.  I  have  been  assured  that  my  privacy  will  be  safeguarded.  I  will  be 
assigned  a  control  or  code  number,  which  thereafter  will  be  the  only 
identifying  entry  on  any  of  the  research  records.  The  Researcher  will 
maintain  the  cross-reference  between  name  and  control  number.  It  will  be 
decoded  only  when  beneficial  to  me  or  if  some  circumstances,  which  is  not 
apparent  at  this  time,  would  make  it  clear  that  decoding  would  enhance  the 
value  of  the  research  data.  In  all  cases,  the  provisions  of  the  Privacy  Act 
Statement  will  be  honored. 

b.  I  understand  that  a  record  of  the  information  contained  in  this 
Consent  Statement  or  derived  from  the  experiment  described  herein  will  be 
retained  permanently  at  the  Naval  Postgraduate  School  or  by  higher 
authority.  I  voluntarily  agree  to  its  disclosure  to  agencies  or  individuals 
indicated  in  paragraph  3  and  I  have  been  informed  that  failure  to  agree  to 
such  disclosure  may  negate  the  purpose  for  which  the  experiment  was 
conducted. 

c.  I  also  understand  that  disclosure  of  the  requested  information, 
including  my  Social  Security  Number,  is  voluntary. 


Name,  Grade/Rank  (if  applicable) 
[Please  print] 


Signature  of  Volunteer 


Date 
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APPENDIX  D.  TASK  DESCRIPTIONS 


Spatial  Orientation 


Background 

Unmanned  air  vehicles  are  nowadays  seen  as  an  area  of  great  importance  in  the 
aerospace  industry.  An  essential  part  of  these  aeronautical  systems  is  the 
ground  control  station  or  GCS.  This  is  the  unit  on  the  ground  that  sends  and 
receives  signals  from  one  or  several  airborne  units.  These  are  normally  very 
complicated  systems  that  require  many  personnel.  There  is  a  general  accepted 
way  of  thinking  that  says  there  are  six  main  instruments,  which  give  an  overall 
picture  of  the  aircrafts  flight  condition.  These  six  primary  instruments  are 
airspeed  indicator,  altimeter,  artificial  horizon,  direction  indicator,  vertical  speed 
indicator  and  turn  and  bank  indicator.  It  is  important  to  group  these  instruments  in 
the  right  way  in  order  to  let  the  pilot  clearly  interpret  the  situation  without 
confusion.  Over  the  years  many  different  layouts  have  been  experimented; 
however  there  is  now  one  accepted  layout,  which  all  modern  airplanes  adopt. 
This  is  known  as  the  basic  T. 


Artificial  Horizon 

This  indicates  the  pitch,  bank  and  heading  attitude  of  the  aircraft.  Pitch,  bank  and 
heading  attitude  are  represented  by  one  moving  element.  This  is  a  surface  that 
symbolizes  the  natural  horizon.  This  moves  in  three  axes  to  indicate  the  change 
in  all  three  parameters  simultaneously.  A  fixed  horizontal  line  on  the  indicator 
represents  the  aircraft. 

You  are  the  operator  of  a  combined  rotor-/fixed-wing  based  Unmanned  Aerial 
Vehicle  (UAV)  of  type  Eagle  Eye. 
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The  Eagle  Eye  has  a 
wingspan  of  15.2  ft,  is 
17.9  ft  in  length,  is  5.7  ft 
high,  and  weighs  around 
2,000  pounds  (depending 
on  payload). 

The  secure  altitude  is 
defined  by  2500  feet. 
Below  this  altitude  the  risk 
of  crashes  and  of  being 
shot  down  by  enemy  fire 
soars. 


Experiment  Goals 

The  goal  of  this  experiment  is  to  test  the  two  different  flight  instrument  (traditional 
-  Weber  box)  setups  regarding  their  capabilities  to  support  the  operator  of 
ground  control  station  in  controlling  the  aircraft.  Of  particular  interest  is  to 
determine  operator  of  increased  workload  and  perception  of  unusual  attitudes. 
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Experiment  Schedule 

You  will  conduct  two  sets  of  trials. 

The  order  in  which  you  will  experience  the  two  sets  randomly  assigned.  The  only 
difference  between  the  sets  is  which  set  of  flight  instruments  is  used  and  the 
increased  workload  while  operating  two  UAVs  simultaneously. 

After  the  task  briefing,  you  will  get  the  opportunity  to  become  familiar  with 
operating  the  flight  simulator  software  before  you  start  with  your  assigned  block 
of  trials.  After  finishing  your  first  block,  you  will  have  a  short  break  to  fill  out  a 
brief  questionnaire,  and  then  you  will  start  with  the  second  block  of  trials. 

After  you  have  finished  both  sets  of  trials  you  will  be  de-briefed  and  asked  to 
respond  to  a  post-experiment  questionnaire. 

Remember: 

•  You  are  a  volunteer  -  we  truly  appreciate  your  time  and  willing  to 
participate  at  our  experiment! 

•  Please  complete  all  trials  if  possible,  but  you  can  quit  the  experiment  at 
any  time  if  you  need  to. 

•  In  case  you  might  experience  discomfort,  please  do  not  hesitate  to  inform 
the  experimenter  and  please  do  not  hesitate  to  abort  the  experiment  if  the 
level  of  discomfort  is  too  high  for  you. 

•  All  personal  data  will  be  handled  confidentially  and  anonymously. 
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Experiment  Schedule  Diagram 
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Task  One 


The  UAV  is  on  a  test  flight  and  enters  a  sector  of  high  crosswinds.  To  simulate 
the  broken  data-link,  you  will  be  exposed  to  an  unknown  attitude  of  the  aircraft. 

Experiment  Implementation 

You  will  operate  the  UAV  for  15  minutes  or  so.  Periodically,  the  experimenter  will 
give  you  the  order  to  judge  the  current  spatial  attitude  of  the  UAV.  The  simulation 
will  freeze  at  this  moment. 


EXPERIMENTER 


You  will  then  hear  the  command  “Stop  Flying.”  The  monitor  will  be  switched  off 
and  immediately  after  switched  on  again,  simulating  data  link  interruption.  You 
will  begin  to  evaluate  your  current  spatial  attitude  using  your  flight  instruments. 
When  you  feel  you  know  what  the  current  spatial  attitude  is  like,  say  loudly 
“STOP.”  At  this  moment  the  flight  instruments  will  be  switched  off. 

Now  you  have  to  express  the  observed  spatial  attitude  of  the  UAV  relative  to  the 
ground.  You  will  do  this  by  determining  pitch  and  roll  which  you  will  draw  on  an 
orientation  chart. 

The  time  to  accomplish  this  task  and  the  accuracy  of  your  spatial  judgment  will 
be  measured. 

You  will  have  one  test  trial  to  become  familiar  with  procedure. 

Do  you  have  any  questions? 
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Task  Two 


The  UAVs  are  on  a  test  flight  and  enter  a  sector  of  bad  weather.  The  data-link 
connection  to  the  UAVs  was  interrupted  because  of  the  weather  conditions 
(simulated  by  switching  off  the  monitor).  When  the  data-link  connection  is 
reestablished,  your  task  is  to  judge  the  current  spatial  attitude  of  the  UAVs.  The 
simulation  will  freeze  at  this  moment. 

Experiment  Implementation 

You  will  “fly”  the  UAVs  for  15  minutes  or  so.  Periodically,  the  experimenter  will 
give  you  the  order  to  judge  the  current  spatial  attitude  of  the  UAVs.  The 
simulation  will  freeze  at  this  moment. 


EXPERIMENTER 


You  will  then  hear  the  command  “Stop  Flying.”  The  Monitor  will  be  switched  off 
after  immediately  switched  on  again,  simulating  data  link  interruption.  You  will 
begin  to  evaluate  your  current  spatial  attitude  using  your  flight  instruments.  When 
you  feel  you  know  what  the  current  spatial  attitude  is  like,  say  loudly  “STOP.”  At 
this  moment  the  flight  instruments  will  be  switched  off. 

Now  you  have  to  express  the  observed  spatial  attitude  of  the  UAVs  relative  to  the 
ground.  You  will  do  this  by  determining  pitch  and  roll  which  you  will  draw  on  an 
orientation  chart. 

The  time  to  accomplish  this  task  and  the  accuracy  of  your  spatial  judgment  will 
be  measured. 

You  will  have  one  test  trial  to  become  familiar  with  procedure. 

Do  you  have  any  questions? 
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APPENDIX  E.  QUESTIONNAIRES 


Initial  Questionnaire 

Date: 


The  information  is  required  for  research  and  your  name  will  not  be  used  therefore  please  answer  as  fully  and 
truthfully  as  possible. 


Participant  ID 


First  Name: 


Middle  Initial: 


Last  Name: 


Age: 


Gender: 


Nationality: 


Male 


□ 


Female  □ 


Are  you  experienced  in  aviation  in  any 
regard  (incl.  playing  Flight  Simulation 
Games)? 

YES  □  NO  □ 

Are  you  an  aviator/pilot? 

YES  □  NO  □ 

Are  you  familiar  with  the  basic  set  of 
flight  instruments? 

YES  □  NO  □ 

Are  you  familiar  with  the  basic  flight 
controls  to  steer  an  airplane/control 

UAV? 

YES  □  NO  □ 

Did  you  operate  any  of  these  aircraft?: 
Fixed-Wing  (Jet/Propeller), 

Rotor-Wing  Aircraft,  UAV 

Fixed  □  Rotor  □  Jet  □ 

Propeller  □  UAV  □ 

How  many  flight  hours  do  you  have? 

Hours 

How  many  hours  in  a  flight  simulator  do 
you  have? 

Hours 

How  long  ago  was  your  last  flight? 

Years  Month(s) 

How  long  ago  was  your  last  use  of  a 
flight  simulator? 

Years  Month(s) 

Are  you  trained  in  procedures  of 
recovery  from  unusual  attitude? 

YES  □  NO 

□ 
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Post-Experiment  Questionnaire 


The  information  is  required  for  research  and  your  name  will  not  be  used  therefore  please  answer  as  fully  and 
truthfully  as  possible. 


I  Participant  ID 


Question 

Answer-Options 

In  general,  how  do  you  judge  the 
difficulty  of  evaluating  the  current 
attitude  of  the  UAV? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very  difficult 

In  general,  how  do  you  judge 
difficulty  of  using  the  traditional  flight 
instruments  for  this  task? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very  difficult 

In  general,  how  do  you  judge  the 
difficulty  of  using  the  WEBER-Box  for 
this  task? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very  difficult 

Did  you  feel  better  aware  of  the 
spatial  orientation  of  the  UAV  with 
the  WEBER-Box  in  comparison  with 
the  traditional  flight  instruments? 

Much  less 

Somewhat 

less 

Border 

line 

Somewhat 

more 

Much  more 

Did  having  the  WEBER-Box  increase 
or  decrease  your  monitoring 
demands  in  comparison  with  the 
traditional  flight  instruments? 

Greatly 

decreased 

Somewhat 

decreased 

Unaffected 

Somewhat 

Increased 

Greatly 

Increased 

Did  having  the  WEBER-Box  increase 
or  decrease  your  overall  workload  in 
comparison  with  the  traditional  flight 
instruments? 

Greatly 

decreased 

Somewhat 

decreased 

Unaffected 

Somewhat 

Increased 

Greatly 

Increased 

Did  having  the  WEBER-Box  increase 
or  decrease  your  overall  frustration 
level  (irritate,  stress,  insecure)  in 
comparison  with  the  traditional  flight 
instruments? 

Greatly 

decreased 

Somewhat 

decreased 

Unaffected 

Somewhat 

Increased 

Greatly 

Increased 

How  would  you  characterize  your 
Computer  Skills 

None 

Fair 

Average 

Good 

Excellent 

How  much  experience  do  you  have 
with  computer  games? 

None 

Fair 

Average 

Good 

Excellent 
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In  General: 


Experiment  Tasks: 


WEBER-Box: 
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APPENDIX  F.  EVALUATION  FORMS 
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Participant  iD:  . 

WEBER  BOX:  YES  □  NO  □ 


Pitch 


Test ... 


Roii 


84 


85 


86 


Pitch  and  Roll  Setups 


Test  A 


Trial 

Pitch 

Roll 

D 

■65 

■150 

C 

■50 

50 

E 

■45 

100 

F 

45 

■65 

A 

■75 

120 

B 

■85 

■30 

Test  B 


Trial 

Pitch 

Roll 

FandA 

45and^75 

■65and120 

EandC 

■45and^50 

100and50 

DandB 

■65and^85 

■150and^30 

AandC 

■75and^50 

120and50 

BandF 

■85and45 

■30and^65 

CandD 

■50and^65 

50and^150 
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A. 


APPENDIX  G.  NATIONAL  AERONAUTICS  AND  SPACE 
ADMINISTRATION  -  TASK  LOAD  INDEX  (NASA-TLX) 
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T<mporal  OcuMnd 
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APPENDIX  H.  EXPERIMENT  DATA 


^JMP  -  [A1LMYTTAS_ALL-S0RTED] 


^  File  Edit  lables  Rows  Cols  DOE  Analyze  Graph  Tools  View  Window  Help 

DDBg@y#Ei  j aii_myttas_all-sorte[ ^ 


-AII_MYTTAS_ALL-SORTE 
»  Source 

i 

Participant 

Age 

Pilot?  (0/1) 

Flight  Hours 

New  Design  (0/1) 

Started  with  Box  (0/1) 

Trial 

Onentation  Time 

Number  of  UAVs  (0/1) 

244 

11 

36 

0 

0 

0 

1 

4 

6.9 

0 

245 

11 

36 

0 

0 

0 

1 

5 

6.8 

0 

246 

11 

36 

0 

0 

0 

1 

6 

6.2 

0 

247 

11 

36 

0 

0 

1 

1 

1 

42 

0 

248 

11 

36 

0 

0 

1 

1 

2 

2.3 

0 

249 

11 

36 

0 

0 

1 

1 

3 

2.8 

0 

250 

11 

36 

0 

0 

1 

1 

4 

2.7 

0 

251 

11 

36 

0 

0 

1 

1 

5 

2.9 

0 

252 

11 

36 

0 

0 

1 

1 

6 

2.7 

0 

253 

11 

36 

0 

0 

0 

1 

1 

5.9 

1 

254 

11 

36 

0 

0 

0 

1 

2 

6.6 

1 

»  Columns  (9/0) 

255 

11 

36 

0 

0 

0 

1 

3 

7.5 

1 

SI  Participant 
@  Age 

256 

11 

36 

0 

0 

0 

1 

4 

6.9 

1 

257 

11 

36 

0 

0 

0 

1 

5 

6.8 

1 

0  Flight  Hours 
il  New  Design  (0/1) 

0  Started  with  Box  (0/1) 

0  Trial 

258 

11 

36 

0 

0 

0 

1 

6 

62 

1 

259 

11 

36 

0 

0 

1 

1 

1 

4.2 

1 

260 

11 

36 

0 

0 

1 

1 

2 

2.3 

1 

261 

11 

36 

0 

0 

1 

1 

3 

2.8 

1 

0  Orientation  Time 

262 

11 

36 

0 

0 

1 

1 

4 

27 

1 

il  Number  of  UAVs  (0/1) 

263 

11 

36 

0 

0 

1 

1 

5 

2.9 

1 

264 

11 

36 

0 

0 

1 

1 

6 

2.7 

1 

265 

101 

34 

1 

450 

0 

0 

1 

8.2 

0 

266 

101 

34 

1 

450 

0 

0 

2 

6.7 

0 

267 

101 

34 

1 

450 

0 

0 

3 

8 

0 

268 

101 

34 

1 

450 

0 

0 

4 

7.4 

0 

269 

101 

34 

1 

450 

0 

0 

5 

5.9 

0 

270 

101 

34 

1 

450 

0 

0 

6 

7.5 

0 

271 

101 

34 

1 

450 

1 

0 

1 

2.9 

0 

272 

101 

34 

1 

450 

1 

0 

2 

3.1 

0 

273 

101 

34 

1 

450 

1 

0 

3 

3.2 

0 

274 

101 

34 

1 

450 

1 

0 

4 

3 

0 

tRows 

275 

101 

34 

1 

450 

1 

0 

5 

3.2 

0 

All  Rows  480 

276 

101 

34 

1 

450 

1 

0 

6 

3.6 

0 

Selected  0 

277 

101 

34 

1 

450 

0 

0 

1 

8.2 

1 

Excluded  0 

Midden  0 

Labelled  0 

278 

101 

34 

1 

450 

0 

0 

2 

6.7 

1 

279 

101 

34 

1 

450 

0 

0 

3 

8 

1 

280 

101 

34 

1 

450 

0 

0 

4 

7.4 

1 

281 

101 

34 

1 

450 

0 

0 

5 

5.9 

1 

282 

101 

34 

1 

450 

0 

0 

6 

7.5 

1 

283 

101 

34 

1 

450 

1 

0 

1 

2.9 

1 

284 

101 

34 

1 

450 

1 

0 

2 

3.1 

1 

285 

101 

34 

1 

450 

1 

0 

3 

3.2 

1 

286 

101 

34 

1 

450 

1 

0 

4 

3 

1 

5fi7 

ini 

1 

Asn 

1 

n 

s 

1  ? 

1 
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